Working with JPA managed entities in Mule applications can be difficult. Since the JPA session is not propagated between message processors, transformers are typically needed to produce an entity from a message’s payload, pass it to a component for processing, then serialize it back to an un-proxied representation for further processing. Transactions have been complicated too. Its difficult to coordinate a transaction between multiple components that are operating with JPA entity payloads. Finally the lack of support for JPA queries makes it difficult to load objects without working with raw SQL and the JDBC transport. Mule Support for JPA Entities The JPA module aims to simplify working with JPA managed entities with Mule. It provides message processors that map to an EntityManager’s methods. The message processors participate in Mule transactions, making it easy to structure JPA transactions within Mule flows. The JPA module also provides a @PersistenceContext implementation. This allows Mule components to participate in JPA transactions. Installing the JPA Module To install the JPA Module you need to click on “Help” followed by “Install New Software…” from Mule Studio. Select the “MuleStudio Cloud Connectors Update Site” from the “Work With” drop-down list then find the “Mule Java Persistence API Module Mule Extension.” This is illustrated below: Fetching JPA Entities JPA query language or criteria queries can be executed using the “query” MP. Supplying a statement to the query will execute the given query and return the results to the next message processor, as illustrated in the following Gist: The queryParameters-ref defines the parameters. In this case the message’s payload as the parameters to the query. The following query illustrates how a Map payload could be used to populate query parameters: The query processor also supports criteria queries by setting the queryParameters-ref to an instance of a CriteriaQuery, as illustrated in the functional test snippet below. CriteriaBuilder criteriaBuilder = entityManager.getCriteriaBuilder(); CriteriaQuery criteriaQuery = criteriaBuilder.createQuery(Dog.class); Root from = criteriaQuery.from(Dog.class); Predicate condition = criteriaBuilder.equal(from.get("name"), "Cujo"); criteriaQuery.where(condition); runFlowWithPayloadAndExpect("testQuery", expectedResults, criteriaQuery); You can use the ”find” MP to load a single object if you know its ID: Transactions and Entity Operations The default behavior of most JPA providers, like Hibernate, is to provide proxies on entity relationships to avoid loading full object graphs into memory. When these objects are detached from the JPA session, however, attempts to access relations in the object will often fail because the proxied session is no longer available. This complicates using JPA is Mule applications as JPA objects pass between message processors and inbetween flows and the session subsequently becomes unavailable. The JPA module allows you to avoid this by wrapping your operations in a transactional block. Let’s first look at how to persist an object then query it within a transaction. The below assumes the message’s payload is an instance of the Dog domain class. Now let’s see how we can use the merge processor to attach a JPA object to a new session. This can be useful when passing a JPA entity from one flow to another. ....other processing here.... Detaching an entity is just as simple: Component Operations with JPA The real power of using JPA with Mule is allowing your business services to participate in Mule managed JPA transactions. A @PersistenceContext EntityManager reference in your component class will cause Mule to inject a reference to a transactional flow’s current EntityManager for that method, as illustrated in the following class: public class DogServiceImpl { @PersistenceContext EntityManager entityManager; public Dog groom(Dog dog) { return entityManager.merge(dog); } } We can now wire the component up in a flow: Conclusion JPA is an important part of the JEE ecosystem and hopefully this module will simplify your use of JPA managed entities in Mule applications.

some time ago at oredev, after the sessions, there was "birds of a feather" - a sort of mini-unconference. anyone could write up a topic on the whiteboard; interested individuals added their names, and each group got allocated a room to chat about the topic. i joined the "spatial indexing" group, and we spent a fascinating hour and a half talking about spatial indexing methods, reminding me of several interesting algorithms and techniques. spatial indexing is increasingly important as more and more data and applications are geospatially-enabled. efficiently querying geospatial data, however, is a considerable challenge: because the data is two-dimensional (or sometimes, more), you can't use standard indexing techniques to query on position. spatial indexes solve this through a variety of techniques. in this post, we'll cover several - quadtrees , geohashes (not to be confused with geohashing ), and space-filling curves - and reveal how they're all interrelated. quadtrees quadtrees are a very straightforward spatial indexing technique. in a quadtree, each node represents a bounding box covering some part of the space being indexed, with the root node covering the entire area. each node is either a leaf node - in which case it contains one or more indexed points, and no children, or it is an internal node, in which case it has exactly four children, one for each quadrant obtained by dividing the area covered in half along both axes - hence the name. a representation of how a quadtree divides an indexed area. source: wikipedia inserting data into a quadtree is simple: starting at the root, determine which quadrant your point occupies. recurse to that node and repeat, until you find a leaf node. then, add your point to that node's list of points. if the list exceeds some pre-determined maximum number of elements, split the node, and move the points into the correct subnodes. a representation of how a quadtree is structured internally. to query a quadtree, starting at the root, examine each child node, and check if it intersects the area being queried for. if it does, recurse into that child node. whenever you encounter a leaf node, examine each entry to see if it intersects with the query area, and return it if it does. note that a quadtree is very regular - it is, in fact, a trie , since the values of the tree nodes do not depend on the data being inserted. a consequence of this is that we can uniquely number our nodes in a straightforward manner: simply number each quadrant in binary (00 for the top left, 10 for the top right, and so forth), and the number for a node is the concatenation of the quadrant numbers for each of its ancestors, starting at the root. using this system, the bottom right node in the sample image would be numbered 11 01. if we define a maximum depth for our tree, then, we can calculate a point's node number without reference to the tree - simply normalize the node's coordinates to an appropriate integer range (for example, 32 bits each), and then interleave the bits from the x and y coordinates -each pair of bits specifies a quadrant in the hypothetical quadtree. geohashes this system might seem familiar: it's a geohash ! at this point, you can actually throw out the quadtree itself - the node number, or geohash, contains all the information we need about its location in the tree. each leaf node in a full-height tree is a complete geohash, and each internal node is represented by the range from its smallest leaf node to its largest one. thus, you can efficiently locate all the points under any internal node by indexing on the geohash by performing a query for everything within the numeric range covered by the desired node. querying once we've thrown away the tree itself becomes a little more complex. instead of refining our search set recursively, we need to construct a search set ahead of time. first, find the smallest prefix (or quadtree node) that completely covers the query area. in the worst case, this may be substantially larger than the actual query area - for example, a small shape in the center of the indexed area that intersects all four quadrants would require selecting the root node for this step. the aim, now, is to construct a set of prefixes that completely covers the query region, while including as little area outside the region as possible. if we had no other constraints, we could simply select the set of leaf nodes that intersect the query area - but that would result in a lot of queries. another constraint, then, is that we want to minimise the number of distinct ranges we have to query for. one approach to doing this is to start by setting a maximum number of ranges we're willing to have. construct a set of ranges, initially populated with the prefix we identified earlier. pick the node in the set that can be subdivided without exceeding the maximum range count and will remove the most unwanted area from the query region. repeat this until there are no ranges in the set that can be further subdivided. finally, examine the resulting set, and join any adjacent ranges, if possible. the diagram below demonstrates how this works for a query on a circular area with a limit of 5 query ranges. how a query for a region is broken into a series of geohash prefixes/ranges. this approach works well, and it allows us to avoid the need to do recursive lookups - the set of range lookups we do execute can all be done in parallel. since each lookup can be expected to require a disk seek, parallelizing our queries allows us to substantially cut down the time required to return the results. still, we can do better. you may notice that all the areas we need to query in the above diagram are adjacent, yet we can only merge two of them (the two in the bottom right of the selected area) into a single range query, requiring us to do 4 separate queries. this is due in part to the order that our geohashing approach 'visits' subregions, working left to right, then top to bottom in each quad. the discontinuity as we go from top right to bottom left quad results in us having to split up some ranges that we could otherwise make contiguous. if we were to visit regions in a different order, perhaps we could minimise or eliminate these discontinuities, resulting in more areas that can be treated as adjacent and fetched with a single query. with an improvement in efficiency like that, we could do fewer queries for the same area covered, or conversely, the same number of queries, but including less extraneous area. illustrates the order in which the geohashing approach 'visits' each quad. hilbert curves suppose instead, we visit regions in a 'u' shape. within each quad, of course, we also visit subquads in the same 'u' shape, but aligned so as to match up with neighbouring quads. if we organise the orientation of these 'u's correctly, we can completely eliminate any discontinuities, and visit the entire area at whatever resolution we choose continuously, fully exploring each region before moving on to the next. not only does this eliminate discontinuities, but it also improves the overall locality. the pattern we get if we do this may look familiar - it's a hilbert curve. hilbert curves are part of a class of one-dimensional fractals known as space-filling curves , so named because they are one dimensional lines that nevertheless fill all available space in a fixed area. they're fairly well known, in part thanks to xkcd's use of them for a map of the internet . as you can see, they're also of use for spatial indexing, since they exhibit exactly the locality and continuity required. for example, if we take another look at the example we used for finding the set of queries required to encompass a circle above, we find that we can reduce the number of queries by one: the small region in the lower left is now contiguous with the region to its right, and whilst the two regions at the bottom are no longer contiguous with each other, the rightmost one is now contiguous with the large area in the upper right. illustrates the order in which a hilbert curve 'visits' each quad. one thing that our elegant new system is lacking, so far, is a way of converting between a pair of (x,y) coordinates and the corresponding position in the hilbert curve. with geohashing it was easy and obvious - just interleave the x and y coordinates - but there's no obvious way to modify that for a hilbert curve. searching the internet, you're likely to come across many descriptions of how hilbert curves are drawn, but few if any descriptions of how to find the position of an arbitrary point. to figure this out, we need to take a closer look at how the hilbert cure can be recursively constructed. the first thing to observe is that although most references to hilbert curves focus on how to draw the curve, this is a distraction from the essential property of the curve, and its importance to us: it's an ordering for points on a plane. if we express a hilbert curve in terms of this ordering, drawing the curve itself becomes trivial - simply a matter of connecting the dots. forget about how to connect adjacent sub-curves, and instead focus on how we can recursively enumerate the points. hilbert curves are all about ordering a set of points on a 2d plane at the root level, enumerating the points is simple: pick a direction and a start point, and proceed around the four quadrants, numbering them 0 to 3. the difficulty is introduced when we want to determine the order we visit the sub-quadrants in while maintaining the overall adjacency property. examination reveals that each of the sub-quadrants' curves is a simple transformation of the original curve: there are only four possible transformations. naturally, this applies recursively to sub-sub quadrants, and so forth. the curve we use for a given quadrant is determined by the curve we used for the square it's in, and the quadrant's position. with a little work, we can construct a table that encapsulates this: suppose we want to use this table to determine the position of a point on a third-level hilbert curve. for the sake of this example, assume our point has coordinates (5,2) starting with the first square on the diagram, find the quadrant your point is in - in this case, it's the upper right quadrant. the first part of our hilbert curve position, then, is 3 (11 in binary). next, we consult the square shown in the inset of square 3 - in this case, it's the second square. repeat the process: which sub-quadrant does our point fall into? here, it's the lower left one, meaning the next part of our position is 1, and the square we should consult next is the second one again. repeating the process one final time, we find our point falls in the upper right sub-sub-quadrant, our final coordinate is 3 (11 in binary). stringing them together, we now know the position of our point on the curve is 110111 binary, or 55. let's be a little more methodical, and write methods to convert between x,y coordinates and hilbert curve positions. first, we need to express our diagram above in terms a computer can understand: hilbert_map = { 'a': {(0, 0): (0, 'd'), (0, 1): (1, 'a'), (1, 0): (3, 'b'), (1, 1): (2, 'a')}, 'b': {(0, 0): (2, 'b'), (0, 1): (1, 'b'), (1, 0): (3, 'a'), (1, 1): (0, 'c')}, 'c': {(0, 0): (2, 'c'), (0, 1): (3, 'd'), (1, 0): (1, 'c'), (1, 1): (0, 'b')}, 'd': {(0, 0): (0, 'a'), (0, 1): (3, 'c'), (1, 0): (1, 'd'), (1, 1): (2, 'd')}, } in the snippet above, each element of 'hilbert_map' corresponds to one of the four squares in the diagram above. to make things easier to follow, i've identified each one with a letter - 'a' is the first square, 'b' the second, and so forth. the value for each square is a dict, mapping x and y coordinates for the (sub-)quadrant to the position along the line (the first part of the value tuple) and the square to use next (the second part of the value tuple). here's how we can use this to translate x and y coordinates into a hilbert curve position: def point_to_hilbert(x, y, order=16): current_square = 'a' position = 0 for i in range(order - 1, -1, -1): position <<= 2 quad_x = 1 if x & (1 << i) else 0 quad_y = 1 if y & (1 << i) else 0 quad_position, current_square = hilbert_map[current_square][(quad_x, quad_y)] position |= quad_position return position the input to this function is the integer x and y coordinates, and the order of the curve. an order 1 curve fills a 2x2 grid, an order 2 curve fills a 4x4 grid, and so forth. our x and y coordinates, then, should be normalized to a range of 0 to 2order-1. the function works by stepping over each bit of the x and y coordinates, starting with the most significant. for each, it determines which (sub-)quadrant the coordinate lies in, by testing the corresponding bit, then fetches the position along the line and the next square to use from the table we defined earlier. the curve position is set as the least significant 2 bits on the position variable, and at the beginning of the next loop, it's left-shifted to make room for the next set of coordinates. let's check that we've written the function correctly by running our example from above through it: >>> point_to_hilbert(5,2,3)55 presto! for a further test, we can use the function to generate a complete list of ordered points for a hilbert curve, then use a spreadsheet to graph them and see if we get a hilbert curve. enter the following expression into an interactive python interpreter: >>> points =[(x, y)for x in range(8)for y in range(8)]>>> sorted_points = sorted(points, key=lambda k: point_to_hilbert(k[0], k[1],3))>>>print'\n'.join('%s,%s'% x for x in sorted_points) take the resulting text, paste it into a file called 'hilbert.csv', open it in your favorite spreadsheet, and instruct it to generate a scatter plot. the result is, of course, a nicely plotted hilbert curve! the inverse of point_to_hilbert is a straightforward reversal of the hilbert_map; implementing it is left as an exercise for the reader. conclusion there you have it - spatial indexing, from quadtrees to geohashes to hilbert curves. one final observation: if you express the ordered sequence of x,y coordinates required to draw a hilbert curve in binary, do you notice anything interesting about the ordering? does it remind you of anything? just to wrap up, a caveat: all of the indexing methods i've described today are only well-suited to indexing points. if you want to index lines, polylines, or polygons, you're probably out of luck with these methods - and so far as i'm aware, the only known algorithm for effectively indexing shapes is the r-tree , an entirely different and more complex beast.

As one of my previous posts already mentioned, I’m in the process of migrating most if not all of my projects to Gradle. I’m very pleased with the fact it’s going a lot smoother than I had anticipated. However I came across an issue: how to get AspectJ compile-time weaving working with Gradle. My use case is quite simple: I’m using Spring’s @Configurable annotation, which either requires load-time or compile-time weaving to work correctly. Load-time weaving isn’t really an option, so compile time weaving it was. There is a plugin but it’s outdated: it doesn’t work with Gradle 1.6. If this was the case with Maven, I’d be screwed/ I would have to write my own plugin. Luckily with Gradle, it was not that difficult. The old plugin relied on the old Gradle 0.9 DSL, so I could use that as a starting point. The instructions required me to define some new configurations and add some things to the javaCompile task. So I started out with this: configurations { ajc aspects } compileJava { sourceCompatibility="1.6" targetCompatibility="1.6" doLast{ ant.taskdef( resource:"org/aspectj/tools/ant/taskdefs/aspectjTaskdefs.properties", classpath: configurations.ajc.asPath) ant.iajc(source:"1.6", target:"1.6", destDir:sourceSets.main.output.classesDir.absolutePath, maxmem:"512m", fork:"true", aspectPath:configurations.aspects.asPath, sourceRootCopyFilter:"**/.svn/*,**/*.java",classpath:configurations.compile.asPath){ sourceroots{ sourceSets.main.java.srcDirs.each{ pathelement(location:it.absolutePath) } } } } } dependencies { ajc "org.aspectj:aspectjtools:1.6.10" compile "org.aspectj:aspectjrt:1.6.10" aspects "org.springframework:spring-aspects:3.2.2.RELEASE" } This, however, did not work. I was greeted by the following exception: can't determine superclass of missing type org.springframework.transaction.interceptor.TransactionAspectSupport It seems like spring-aspects contains multiple aspects, each requiring some dependencies which are required regardless whether you want to use those aspects. This, by the way, was also required when using the Maven AspectJ plugin. After some POM fishing, I found out I had to add a couple of libraries. So now we have the following dependencies: dependencies { ajc "org.aspectj:aspectjtools:1.6.10" compile "org.aspectj:aspectjrt:1.6.10" aspects "org.springframework:spring-aspects:3.2.2.RELEASE" aspects "javax.persistence:persistence-api:1.0" aspects "org.springframework:spring-tx:3.2.2.RELEASE" aspects "org.springframework:spring-orm:3.2.2.RELEASE" } My code compiled and my aspects were weaved in! Unfortunately, it wasn’t the end of the story. My tests failed. It was missing the aspect library at runtime, as the (now weaved) classes referenced some classes that were contained in spring-aspects and I was subclassing them in my testcode. The quick solution would be to duplicate the spring-aspects dependency in the compile configuration, but that was something I didn’t like. I did however came up with a cleaner solution: I had to make sure the aspect dependencies were added to the compile configuration. A small change in the configurations section was all I needed to do. configurations { ajc aspects compile { extendsFrom aspects } } Hurrah! I wished. Unfortunately now the dependencies I had to add to the build in order to make the weaving work were now also added to the compile configuration. I didn’t need them or used them, so I had to find a way to exclude them from the compile configuration, but still include them in the aspect weaving. The solution was simple and clean: add a new configuration and include those configuration’s dependencies in the weaving classpath. The new configiration is not added in the compile configuration, so your build is not dependent on those dependencies. The final product looked like this: configurations { ajc aspects aspectCompile compile { extendsFrom aspects } } compileJava { sourceCompatibility="1.6" targetCompatibility="1.6" doLast{ ant.taskdef( resource:"org/aspectj/tools/ant/taskdefs/aspectjTaskdefs.properties", classpath: configurations.ajc.asPath) ant.iajc(source:"1.6", target:"1.6", destDir:sourceSets.main.output.classesDir.absolutePath, maxmem:"512m", fork:"true", aspectPath:configurations.aspects.asPath}, sourceRootCopyFilter:"**/.svn/*,**/*.java",classpath:"${configurations.compile.asPath};${configurations.aspectCompile.asPath}"){ sourceroots{ sourceSets.main.java.srcDirs.each{ pathelement(location:it.absolutePath) } } } } } dependencies { ajc "org.aspectj:aspectjtools:1.6.10" compile "org.aspectj:aspectjrt:1.6.10" aspects "org.springframework:spring-aspects:3.2.2.RELEASE" aspectCompile "javax.persistence:persistence-api:1.0" aspectCompile "org.springframework:spring-tx:3.2.2.RELEASE" aspectCompile "org.springframework:spring-orm:3.2.2.RELEASE" } The entire process took me about an hour, although most of it was looking up stuff in the Gradle documentation. I’m also really impressed with the error notification of Gradle. If something goes wrong, it’s quite accurate in pinpointing the problem. This was one of the more complex things in the builds I have, so anything else should be a breeze. One thing I really, really like about Gradle is that you can merge all build configuration of a multi-module project into a single build.gradle file. For a build with 10 modules this meant going from maintaining 11 Maven POM files to only one gradle build script. Even better: the single build.gradle file is 3 times smaller than the projects root POM! If that’s not nice, I don’t know what is.

Quick reference for converting Java objects to various formats (byte array, JSON, XML) and back, using different libraries for serialization and deserialization.

When posting source code to my blog, I often need to convert less than signs (<), and greater than signs (>) to their respective entity references so that they are not confused as HTML tags when the browser renders the output. I have often done this using quick search-and-replace syntax like%s//\>/g with vim or Perl. However, Groovy 2.1 introduced a method to do this and in this post I demonstrate a Groovy script that makes use of thatgroovy.xml.XmlUtil.escapeXml(String) method. escapeXml.groovy #!/usr/bin/env groovy /* * escapeXml.groovy * * Requires Groovy 2.1 or later. */ if (args.length < 1) { println "USAGE: groovy escapeXml.groovy " System.exit(-1) } def inputFileName = args[0] println "Processing ${inputFileName}..." def inputFile = new File(inputFileName) String outputFileName = inputFileName + ".escaped" def outputFile = new File(outputFileName) if (outputFile.createNewFile()) { outputFile.text = groovy.xml.XmlUtil.escapeXml(inputFile.text) } else { println "Unable to create file ${outputFileName}" } The XmlUtil.escapeXml method is intended to, as its GroovyDoc states, "escape the following characters " ' & < > with their XML entities." Running source code through it helps to convert symbols to XML entity references that will be rendered properly by the browser. This is particularly helpful with Java code that uses generics, for example. The Groovydoc states that the following transformations from symbols to corresponding entity references are supported: Symbol Entity Reference " " ' ' & & < < > > One of the advantages of this approach is that I can escape all five of these special symbols in an entire String or file with a single command rather than one symbol at a time. The Groovydoc for this XmlUtil.escapeXml method also states things that this method does not do: "Does not escape control characters" [use XmlUtil.escapeControlCharacters(String) for this] "Does not support DTDs or external entities" "Does not treat surrogate pairs specially" "Does not perform Unicode validation on its input" My example above showed a Groovy script file that makes use of XmlUtil.escapeXml(String), but it can also be run inline on the command-line. This is done in DOS, for example, as shown here: type escapeXml.groovy | groovy -e "println groovy.xml.XmlUtil.escapeXml(System.in.text)" That command just shown will take the provided file (escapeXml.groovy itself in this case) and render output with the specific symbols replaced with entity references. It could be handled the same way in Linux/Unix with "cat" rather than "type." This is shown in the next screen snapshot. This blog post has shown how XmlUtil.escapeXml(String) can be used within a script or on the command-line to escape certain commonly problematic XML characters to their entity references. Although not shown here, one could embed such code within a Java application as well.

zipf’s law states that the frequency of a word in a corpus of text is proportional to it’s rank – first noticed in the 1930′s. unlike a “law” in the sense of mathematics or physics, this is purely on observation, without strong explanation that i can find of the causes. we can explore this concept fairly simply on a bit of text using nltk , which provides handy apis for accessing and processing text. there is a good textbook on the subject, “ natural language processing with python ” ( read my review ), with lots of motivating examples like this. import nltk from nltk.corpus import reuters from nltk.corpus import wordnet reuters_words = [w.lower() for w in reuters.words()] words = set(reuters_words) counts = [(w, reuters_words.count(w)) for w in words] the first step of counting word frequencies takes quite a while (an hour or two). here are the top tokens (obviously not all words) >>> [(w, c) for (w, c) in counts if c > 5000] [('.', 94687), ('s', 15680), ('with', 6179), ("'", 11272), ('>', 7449), ('year', 7529), ('000', 10277), ('loss', 5124), ('u', 6392), ('pct', 9810), ('"', 6816), ('from', 8217), ('for', 13782), ('2', 6528), ('at', 7017), ('be', 6357), ('the', 69277), (';', 8762), ('he', 5215), ('net', 6989), ('is', 7668), ('it', 11104), ('in', 29253), ('billion', 5829), ('lt', 8696), ('-', 13705), ('of', 36779), ('&', 8698), ('to', 36400), ('vs', 14341), ('was', 5816), ('1', 9977), ('and', 25648), ('dlrs', 12417), ('by', 7101), ('its', 7402), ('mln', 18623), ('cts', 8361), ('on', 9244), ('that', 7540), ('3', 5091), ('a', 25103), (',', 72360), ('said', 25383), ('will', 5952)] next, i joined this to wordnet data – while exploring zipf’s law, i want to test the hypthesis that common words are more likely to be irregular. we also generate the rankings of each word in the set, but frequencies, and by number of “synsets” in wordnet (synsets are sets meanings) import scipy.stats as ss amb = [(w, c, len(wordnet.synsets(w))) / for (w, c) in counts if len(wordnet.synsets(w)) > 0] amb_p_rank = ss.rankdata([p for (w, c, p) in amb]) amb_c_rank = ss.rankdata([c for (w, c, p) in amb]) amb_ranked = zip(amb, amb_p_rank, amb_c_rank) amb_ranked[100:110] out[37]: [('regulator', 2, 3, 8945.0, 5500.0), ('friend', 2, 5, 12344.0, 5500.0), ('feeling', 19, 19, 16810.0, 12979.0), ('sustaining', 7, 7, 14215.0, 10282.5), ('spectrum', 8, 2, 6142.0, 10684.5), ('consenting', 1, 2, 6142.0, 2218.5), ('resignations', 3, 3, 8945.0, 7215.5), ('dozen', 11, 2, 6142.0, 11554.0), ('affairs', 75, 5, 12344.0, 15397.5), ('mostly', 57, 2, 6142.0, 15038.0)] the last line combines the data values together into one big list of tuples, for convenience. to do a quick test of zipf’s law, we can test that the rank correlates to the log of the count: numpy.corrcoef(amb_c_rank, [math.log(c) for (w, c, p) in amb]) out[106]: array([[ 1. , 0.95322962], [ 0.95322962, 1. ]]) we can also demonstrate this a different way. first, we sort the records by occurence: amb_ranked_sorted = sorted(amb_ranked, key=lambda (w, c, p, cr, pr): c) and then we take some samples. note how when we move by approximately 3x rank, the frequency goes down by about 1/3 (this will vary based on how the rank is counted, especially at the long tail where there are many matching counts in a row) amb_ranked_sorted[-50] out[121]: ('profit', 2960, 4, 10907.0, 17128.0) amb_ranked_sorted[-150] out[122]: ('major', 1000, 13, 16262.0, 17028.0) amb_ranked_sorted[-450] out[123]: ('goods', 408, 4, 10907.0, 16728.0) amb_ranked_sorted[-1350] out[124]: ('hope', 113, 9, 15215.0, 15827.5) finally, we can plot this: import matplotlib rev = [l-r+1 for r in amb_c_rank] plt.plot([math.log(c) for c in rev], [math.log(c) for (w, c, p) in amb], 'ro') this isn’t exactly straight, especially at the end, likely due to how ranks are computed, but it’s close. now we can plot the number of “meanings” in wordnet vs use in the reuters corpus: plt.plot([c for (w,c,p) in amb], [p for (w,c,p) in amb], 'bs') out[150]: [] plt.show() the problem is that there is no notion of context, for instance the common word “at” is only referenced by the chemical element “astatine”: wordnet.synsets('at') out[149]: [synset('astatine.n.01'), synset('at.n.02')] thus, a better technique is to look at verbs only. def wordnet_verbs(w): synsets = wordnet.synsets(w) verbs = [w for w in synsets if w.pos == 'v'] return verbs amb_v = [(w, c, len(wordnet_verbs(w)), len(wordnet.synsets(w))) \ for (w, c) in counts if len(wordnet_verbs(w)) > 0] amb_v[100:110] out[167]: [('shoots', 1, 20, 22), ('suffice', 1, 1, 1), ('acquainting', 1, 3, 3), ('perfumes', 1, 2, 4), ('safeguard', 14, 2, 4), ('arrays', 4, 2, 6), ('crowns', 143, 4, 16), ('roll', 21, 18, 33), ('intend', 63, 4, 4), ('palms', 2, 1, 5)] plt.plot([c for (w,c,vc,wc) in amb_v], [vc for (w,c,vc,wc) in amb_v], 'bs',\ [c for (w,c,vc, wc) in amb_v], [wc for (w,c,vc,wc) in amb_v], 'ro') and, we can plot these (y axis is # of alternate meanings, x axis is frequency, blue is counts of verb meanings, and red is counts of word meanings). there does seem to be a pattern here, but not what i expect – i would expect some straight-line upwards, with alternate meanings increasing with usage, which is actually the opposite. this likely means that the wordnet thesaurus counts are a poor indicator of irregularity. rather, words which have many meanings are discouraged in news writing, as they encourage ambiguity. for a final note, lets look at what are these common words are: [(w,c,vc,wc) for (w,c,vc,wc) in amb_v if vc > 40] out[179]: [('cut', 905, 41, 70), ('makes', 169, 49, 51), ('runs', 38, 41, 57), ('making', 369, 49, 52), ('gave', 208, 44, 44), ('breaks', 14, 59, 75), ('take', 745, 42, 44), ('broke', 40, 59, 60), ('run', 125, 41, 57), ('give', 303, 44, 45), ('cuts', 319, 41, 61), ('broken', 36, 59, 72), ('giving', 99, 44, 48), ('took', 140, 42, 42), ('breaking', 19, 59, 60), ('takes', 91, 42, 44), ('taken', 244, 42, 44), ('make', 592, 49, 51), ('taking', 197, 42, 44), ('given', 370, 44, 47), ('gives', 47, 44, 45), ('made', 987, 49, 52), ('break', 76, 59, 75), ('giveing', 1, 44, 44), ('cutting', 122, 41, 54), ('running', 73, 41, 52), ('ran', 29, 41, 41)] “cut” has a staggering 70 entrieds in wordnet – many, but not all verbs. “has”, by contrast, has merely twenty.

Recently a respected member of the Apache community tried Log4j 2 and wrote on Twitter: (Quote from Mark Struberg: @TheASF #log4j2 rocks big times! Performance is close to insane ^^ http://logging.apache.org/log4j/2.x/ ) It happened shortly after Remko Popma contributed something which is now called the “AsyncLoggers”. Some of you might know Log4j 2 has AsyncAppenders already. They are similar like the ones you can find in Log4j 1 and other logging frameworks. I am honest: I wasn’t so excited about the new feature until I read the tweet on its performance and became curious. Clearly Java logging has many goals. Among them: logging must be as fast as hell. Nobody wants his logging framework to become a bottleneck. Of course you’ll always have a cost when logging. There is some operation the CPU must perform. Something is happening, even when you decide NOT to write a log statement. Logging is expected to be invisible. Until now, the well-known logging frameworks were similar in speed. Benchmarks are unreliable after all. We have made some benchmarks over at Apache Logging. Sometimes one logging frameworks wins, sometimes the other. But at the end of the day you can say they are all very good and you can choose whatever your liking is. Until we got Remko’s contribution and Log4j 2 became “insanely fast”. Small software projects running one thread might not care about performance so much. When running a SaaS you simply don’t know when your app gets so much attraction that you need to scale. Then you suddenly need some extra power. With Log4j 2, running 64 threads might bring you twelve times more logging throughput than with comparable frameworks. We speak of more than 18,000,000 messages per second, while others do around 1,500,000 or less in the same environment. I saw the chart, but simply couldn’t believe it. There must be something wrong. I rechecked. I ran the tests myself. It’s like that: Log4j 2 is insanely fast. Async Performance, last read on July 19, 2013 As of now, we have a logging framework which performs lots better than every other logging framework out there. As of now we need to justify our decision when we do not want to use Log4j 2, if speed matters.Everything else than Log4j 2 can become a bottleneck and a risk. With such a fast logging framework you might even consider to log a bit more in production than you did before. Eventually I wrote Remko an e-mail and asked him what exactly the difference between the old AsyncAppenders and the new Asynchronous Loggers is. The difference between old AsynAppenders and new AsyncLoggers “The Asynchronous Loggers do two things differently than the AsyncAppender”, he told me, “they try to do the minimum amount of work before handing off the log message to another thread, and they use a different mechanism to pass information between the producer and consumer threads. AsyncAppender uses an ArrayBlockingQueue to hand off the messages to the thread that writes to disk, and Asynchronous Loggers use the LMAX Disruptor library. Especially the Disruptor has made a large performance difference.” In other terms, the AsyncAppender use a first-in-first-out Queue to work through messages. But the Async Logger uses something new – the Disruptor. To be honest, I had never heard of it. And furthermore, I never thought much about scaling my logging framework. When somebody said “scale the system”, I thought about the database, the app server and much more, but usually not logging. In production, logging was off. End of story. But Remko thinks about scaling when it comes to logging. “Looking at the performance test results for the Asynchronous Loggers, the first thing you notice is that some ways of logging scale much better than others. By scaling better I mean that you get more throughput when you add more threads. If your throughput increases a constant amount with every thread you add, you have linear scalability. This is very desirable but can be difficult to achieve.”, he wrote me. “Comparing synchronous to asynchronous, you would expect any asynchronous mechanism to scale much better than synchronous logging because you don’t do the I/O in the producing thread any more, and we all know that ‘I/O is slow’ (and I’ll get back to this in a bit)”. Yes, exactly my understanding. I thought it would be enough to send something to a queue, and something else would pick it up and write the message. The app would go on. This is exactly what the old AsyncAppender does, wrote Remko: “With AsyncAppender, all your application thread needs to do is create a LogEvent object and put it on the ArrayBlockingQueue; the consuming thread will then take these events off the queue and do all the time-consuming work. That is, the work of turning the event into bytes and writing these bytes to the I/O device. Since the application threads do not need to do the I/O, you would expect this to scale better, meaning adding threads will allow you to log more events.” If you believed that like me, take a seat and a deep breath. We were wrong. “What may surprise you is that this is not the case.”, he wrote. “If you look at the performance numbers for the AsyncAppenders of all logging frameworks, you’ll see that every time you double the number of threads, your throughput per thread roughly halves.” “So your total throughput remains more or less flat! AsyncAppenders are faster than synchronous logging, but they are similar in the sense that neither of them gives you more total throughput when you add more threads.”, he told me. It hit me like a hammer. Basically instead of making your logging faster with adding more threads you made basically: nothing. After all Appenders didn’t scale until now. I asked Remko why this was the case. “It turns out that queues are not the most optimal data structure to pass information between threads. The concurrent queues that are part of the standard Java libraries use locks to make sure that values don’t get corrupted and to ensure data visibility between threads.”. LMAX Disruptor? “The LMAX team did a lot of research on this and found that these queues have a lot of lock contention. An interesting thing they found is that queues are always either full or empty: If your producer is faster, your queue will be full most of the time (and that may be a problem in itself ). If your consumer is fast enough, your queue will be empty most of the time. Either way, you will have contention on the head or on the tail of the queue, where both the producer and the consumer thread want to update the same field. To resolve this, the LMAX team came up with the Disruptor library, which is a lock-free data structure for passing messages between threads. Here is a performance comparison between the Disruptor and ArrayBlockingQueue:Performance Comparison.” Wow. After all these years of Java programming I actually felt a bit like a Junior programmer again. I missed the LMAX disruptor and even never considered it a performance problem to use the Queue. I wonder what other performance problems I did not discover so far. I realized, I had to re-learn Java. I asked Remko how he could find a library like the LMAX disruptor. I mean nobody writes software, creates an instance of a Queue-class, doubts its performance and finally searches the internet for “something better”. Or are there really people of that kind? “How I found about the Disruptor? The short answer is, it was all a mistake.”, he started. “Okay, perhaps that was a bit too short, so here is the longer answer: a colleague of mine wrote a small logger, essentially adding a time-stamped log message to a queue, with a background thread that took these strings off the queue and wrote them to disk. He did this because he needed better performance than what he could get with log4j-1.x. I did some testing and found it was faster, I don’t remember exactly by how much. I was quite surprised because I had been using log4j for years and had never thought it would be easily outperformed. Until then I had assumed that the well-known libraries would be fast, because, well… To be honest, I had just assumed. So this was a bit of an eye-opener for me. However, the custom logger was a bit bare-bones in terms of functionality so I started to look around for alternatives.” “Before I start talking about the Disruptor, I have to confess something. I recently went back to see how much faster the custom logger was than log4j-1.x, but when I measured it it was actually slower! It turned out that I had been comparing the custom logger to an old beta of log4j-2.0, I think beta3 or beta4. AsyncAppender in those betas still had a performance issue (LOG4J2-153 if you’re curious). If I had compared the custom logger to the AsyncAppender in log4j-1.x, I would have found that log4j-1.x was faster and I would not have thought about it further. But because I made this mistake I started to look for other high-performance logging libraries that were richer in functionality. I did not find such a logging library, but I ran into a whole bunch of other interesting stuff, including the Disruptor. Eventually I decided to try to combine Log4j-2, which has a very nice code base, with the Disruptor. The result of this was eventually accepted into Log4j-2 itself, and the rest, as they say, was history.” “One thing I came across that I should mention here is Peter Lawrey’sChronicle library. Chronicle uses memory-mapped files to write tens of millions of messages per second to disk with very low latency. Remember that above I said that “we all know that I/O is slow”? Chronicle shows that synchronous I/O can be very, very fast.“. “It was via Peter’s work that I came across the Disruptor. There is a lot of good material out there about the Disruptor. Just to give you a few pointers: Martin Fowler: LMAX Trisha Lee on LMAX under the hood (slightly outdated now but the most detailed material I know of) …and video presentations like this The Disruptor google group is also highly recommended. Recommended readings on Java performance in general are: Martin Thompson’s “Mechanical Sympathy” Martin Thompson Presentations. Martin Thompson has done a number of articles and presentations on various aspects of high performance computing in java. He does a great job of making the complex stuff that is going on under the hood accessible.” My bookmarks folder went full after reading this e-mail, and I appreciate the lots of starting points for improving my knowledge on Java performance. Should I use AsyncLoggers by default? I was sure I want to use the new Async Loggers. This all sounds just fantastic. But on the other hand, I am a bit scared and even a little paranoid to include new dependencies or new technologies like the new Log4j 2 Async Loggers. I asked Remko if he would use the new feature by default or if he would enable them just for a few, limited use cases. “I use Async Loggers by default, yes.”, he wrote me. “One use case when you would _not_ want to use asynchronous logging is when you use logging for audit purposes. In that case a logging error is a problem that your application needs to know about and deal with. I believe that most applications are different, in that they don’t care too much about logging errors. Most applications don’t want to stop if a logging exception occurs, in fact, they don’t even want to know about it. By default, appenders in Log4j-2.0 will suppress exceptions so the application doesn’t need to try/catch every log statement. If that is your usage, then you will not lose anything by using asynchronous loggers, so you get only the benefits, which is improved performance.” “One nice little detail I should mention is that both Async Loggers and Async Appenders fix something that has always bothered me in Log4j-1.x, which is that they will flush the buffer after logging the last event in the queue. With Log4j-1.x, if you used buffered I/O, you often could not see the last few log events, as they were still stuck in the memory buffer. Your only option was setting immediateFlush to true, which forces disk I/O on every single log event and has a performance impact. With Async Loggers and Appenders in Log4j-2.0 your log statements are all flushed to disk, so they are always visible, but this happens in a very efficient manner.” Isn’t it risky to log to use Log4js AsyncLoggers? But considering that Log4j-1 had serious threading issues and the modern world uses cloud computing and clustering all the time to scale their apps,isn’t asynchronous logging some kind of additional risk? Or is it safe? I knew my questions would sound like the questions of a decision maker, not of an developer. But the whole LMAX thing was so new to me and since I maintain the old and really ugly Log4j 1 code, I simply had to ask. Remko: “There are a number of questions in there. First, is Log4j-2 safer from a concurrency perspective than Log4j-1.x? I believe so. The Log4j-2 team has put in considerable effort to support multi-threaded applications, and the asynchronous loggers are just a very recent and relatively small addition to the project. Log4j-2 uses more granular locking than log4j-1.x, and is architecturally simpler, which should result in fewer issues, and any issues that do come up will be easier to fix.” “On the other hand, Log4j-2 is still in beta and is under active development, although recently I think most effort is being spent on fixing things and tying up loose ends rather than adding new features. I believe it is stable enough for production use. If you are considering using Log4j-2, for performance or other reasons, I’d suggest you do your due diligence and test, just like you would before adopting any other 3rd party library in your project.” (Sidenote: A stable version of Log4j2 can be expected soon, most likely autumn 2013). Sounded good to me. And yes, I can perfectly agree with that from my own observations on the project, though I personally did not write code in the Log4j 2 repository. “The other question I see is: Is asynchronous logging riskier than synchronous logging? I don’t think so, in fact, if your application is multi threaded the opposite may be the case: once the log event has been handed off to the consumer thread that does the I/O, there is only that one thread dealing with the layouts, appenders and all the other logging-related components. So after the hand-off you’re single-threaded and you don’t need to worry about any threading issues like deadlock and liveliness etc any more.” “You can take this one step further and make your business logic completely single-threaded, using the disruptor for all I/O or communication with external systems. Single-threaded business logic without lock contention can be blazingly fast. The results at LMAX (6 million transactions/sec, with less than 10 ms latency) speak for themselves.” Reading Remko’s message I learned three things. First, I had to learn more about Java performance. Second, I definitely want to make my applications use Log4j 2. As first step, I will enable it in my Struts 2 apps, which I use often. Third, a web application framework using the LMAX Disruptor might blow us all away. I would like to give a big thank you and a hug to Remko Popma for answering my questions and working on this blog post with me. All errors are my own.

In my previous post quite sometime back I had written about Internal DSLs using Java. In the book Domain Specific Languages by Martin Fowler, he discusses about another type of DSL called external DSLs in which the DSL is written in another language which is then parsed by the host language to populate the semantic model. In the previous example I was discussing about creating a DSL for defining a graph. The advantage of using an external dsl is that any change in the graph data would not require recompilation of the program instead the program can just load the external dsl, create a parse tree and then populate the semantic model. The semantic model will remain the same and the advantage of using the semantic model is that one can make modification to the DSL without making much changes to the semantic model. In the example between Internal DSLs and external DSLs I have not modified the semantic model. To create an external DSL I am making use of ANTLR. What is ANTLR? The definition as given on the official site is: ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files. It’s widely used to build languages, tools, and frameworks. From a grammar, ANTLR generates a parser that can build and walk parse trees. The notable features of ANTLR from the above definition are: parser generator for structured text or binary files can build and walk parse trees Semantic Model In this example I will exploit the above features of ANTLR to parse a DSL and then walk through the parse tree to populate the Semantic model. To recap, the semantic model consists of Graph, Edge and Vertex classes which represent a Graph and an Edge and a Vertex of the Graph respectively. The below code shows the class definitions: public class Graph { private List edges; private Set vertices; public Graph() { edges = new ArrayList<>(); vertices = new TreeSet<>(); } public void addEdge(Edge edge){ getEdges().add(edge); getVertices().add(edge.getFromVertex()); getVertices().add(edge.getToVertex()); } public void addVertice(Vertex v){ getVertices().add(v); } public List getEdges() { return edges; } public Set getVertices() { return vertices; } public static void printGraph(Graph g){ System.out.println("Vertices..."); for (Vertex v : g.getVertices()) { System.out.print(v.getLabel() + " "); } System.out.println(""); System.out.println("Edges..."); for (Edge e : g.getEdges()) { System.out.println(e); } } } public class Edge { private Vertex fromVertex; private Vertex toVertex; private Double weight; public Edge() { } public Edge(Vertex fromVertex, Vertex toVertex, Double weight) { this.fromVertex = fromVertex; this.toVertex = toVertex; this.weight = weight; } @Override public String toString() { return fromVertex.getLabel() + " to " + toVertex.getLabel() + " with weight " + getWeight(); } public Vertex getFromVertex() { return fromVertex; } public void setFromVertex(Vertex fromVertex) { this.fromVertex = fromVertex; } public Vertex getToVertex() { return toVertex; } public void setToVertex(Vertex toVertex) { this.toVertex = toVertex; } public Double getWeight() { return weight; } public void setWeight(Double weight) { this.weight = weight; } } public class Vertex implements Comparable { private String label; public Vertex(String label) { this.label = label.toUpperCase(); } @Override public int compareTo(Vertex o) { return (this.getLabel().compareTo(o.getLabel())); } public String getLabel() { return label; } public void setLabel(String label) { this.label = label; } } Creating the DSL Lets come up with the structure of the language before going into creating grammar rules. The structure which I am planning to come up is something like: Graph { A -> B (10) B -> C (20) D -> E (30) } Each line in the Graph block represents an edge and the vertices involved in the edge and the value in the braces represent the weight of the edge. One limitation which I am enforcing is that the Graph cannot have dangling vertices i.e vertices which are not part of any edge. This limitation can be removed by slightly changing the grammar, but I would leave that as an exercise for the readers of this post. The first task in creating the DSL is to define the grammar rules. These are the rules which your lexer and parser will use to convert the DSL into a Abstract Syntax tree/parse tree. ANTLR then makes use of this grammar to generate the Parser, Lexer and a Listener which are nothing but java classes extending/implementing some classes from the ANTLR library. The creators of the DSL must make use of these java classes to load the external DSL, parse it and then using the listener populate the semantic model as and when the parser encounters certain nodes (think of this as a variant of SAX parser for XML) Now that we know in very brief what ANTLR can do and the steps in using ANTLR, we would have to setup ANTLR i.e download ANTLR API jar and setup up some scripts for generating the parser and lexer and then trying out the language via the command line tool. For that please visit this official tutorial from ANTLR which shows how to setup ANTLR and a simple Hello World example. Grammar for the DSL Now that you have ANTLR setup let me dive into the grammar for my DSL: grammar Graph; graph: 'Graph {' edge+ '}'; vertex: ID; edge: vertex '->' vertex '(' NUM ')' ; ID: [a-zA-Z]+; NUM: [0-9]+; WS: [ \t\r\n]+ -> skip; Lets go rule: graph: 'Graph {' edge+ '}'; The above grammar rule which is the start rule says that the language should start with ‘Graph {‘ and end with ‘}’ and has to contain at lease one edge or more than one edge. vertex: ID; edge: vertex '->' vertex '(' NUM ')' ; ID: [a-zA-Z]+; NUM: [0-9]+; The above four rules say that a vertex should have atleast one character or more than one character. And an edge is defined as collection of two vertices separated by a ‘->’ and with the some digits in the ‘()’. I have named the grammar language as “Graph” and hence once we use ANTLR to generate the java classes i.e parser and lexer we will end up seeing the following classes: GraphParser, GraphLexer, GraphListener and GraphBaseListener. The first two classes deal with the generation of parse tree and the last two classes deal with the parse tree walk through. GraphListener is an interface which contains all the methods for dealing with the parse tree i.e dealing with events such as entering a rule, exiting a rule, visiting a terminal node and in addition to these it contains methods for dealing with events related to entering the graph rule, entering the edge rule and entering the vertex rule. We will be making use of these methods to intercept the data present in the dsl and then populate the semantic model. Populating the semantic model I have created a file graph.gr in the resource package which contains the DSL for populating the graph. As the files in the resource package are available to the ClassLoader at runtime, we can use the ClassLoader to read the DSL script and then pass it on to the Lexer and parser classes. The DSL script used is: Graph { A -> B (10) B -> C (20) D -> E (30) A -> E (12) B -> D (8) } And the code which loads the DSL and populates the semantic model: //Please resolve the imports for the classes used. public class GraphDslAntlrSample { public static void main(String[] args) throws IOException { //Reading the DSL script InputStream is = ClassLoader.getSystemResourceAsStream("resources/graph.gr"); //Loading the DSL script into the ANTLR stream. CharStream cs = new ANTLRInputStream(is); //Passing the input to the lexer to create tokens GraphLexer lexer = new GraphLexer(cs); CommonTokenStream tokens = new CommonTokenStream(lexer); //Passing the tokens to the parser to create the parse trea. GraphParser parser = new GraphParser(tokens); //Semantic model to be populated Graph g = new Graph(); //Adding the listener to facilitate walking through parse tree. parser.addParseListener(new MyGraphBaseListener(g)); //invoking the parser. parser.graph(); Graph.printGraph(g); } } /** * Listener used for walking through the parse tree. */ class MyGraphBaseListener extends GraphBaseListener { Graph g; public MyGraphBaseListener(Graph g) { this.g = g; } @Override public void exitEdge(GraphParser.EdgeContext ctx) { //Once the edge rule is exited the data required for the edge i.e //vertices and the weight would be available in the EdgeContext //and the same can be used to populate the semantic model Vertex fromVertex = new Vertex(ctx.vertex(0).ID().getText()); Vertex toVertex = new Vertex(ctx.vertex(1).ID().getText()); double weight = Double.parseDouble(ctx.NUM().getText()); Edge e = new Edge(fromVertex, toVertex, weight); g.addEdge(e); } } And the output when the above would be executed would be: Vertices... A B C D E Edges... A to B with weight 10.0 B to C with weight 20.0 D to E with weight 30.0 A to E with weight 12.0 B to D with weight 8.0 To summarize, this post creates a external DSL for populating the data for graphs by making use of ANTLR. I will enhance this simple DSL and expose it as an utility which can be used by programmers working on graphs. The post is very heavy on concepts and code, feel free to drop in any queries you have so that I can try to address them for benefit of others as well.

An overview starting with some basic classes of the Java 8 package: Instant, LocalDate, LocalTime, and LocalDateTime.

This technical tip explains how to read outlook email template File (.OFT). Outlook templates are very useful when you want to send a similar email message again and again. Instead of preparing the message from scratch each time, first prepare the message in Outlook and save it as an Outlook template (OFT). After that, whenever you need to send the message, you can create it from the template, saving time writing the same text in the body or the subject line, setting formatting and so on. Aspose.Email’s MailMessage class can be used to load and read an Outlook template (OFT) file. Once the Outlook template is loaded in an instance of the MailMessage class, you can update the sender, recipient, body, subject and other properties. //[C#] // Load the Outlook template (OFT) file in MailMessage's instance MailMessage message = MailMessage.Load("invitation to meeting.oft", MessageFormat.Msg); // Set the sender and recipients information string senderDisplayName = "John"; string senderEmailAddress = "[email protected]"; string recipientDisplayName = "William"; string recipientEmailAddress = "[email protected]"; message.Sender = new MailAddress(senderEmailAddress, senderDisplayName); message.To.Add(new MailAddress(recipientEmailAddress, recipientDisplayName)); message.HtmlBody = message.HtmlBody.Replace("DisplayName", "" + recipientDisplayName + ""); // Set the name, location and time in email body string meetingLocation = "" + "Hall 1, Convention Center, New York, USA" + ""; string meetingTime = "" + "Monday, June 28, 2010" + ""; message.HtmlBody = message.HtmlBody.Replace("MeetingPlace", meetingLocation); message.HtmlBody = message.HtmlBody.Replace("MeetingTime", meetingTime); // Send the email or save as MSG and open in Outlook for further editing SmtpClient client = new SmtpClient("host", 25, "username", "password"); client.Send(message); // Save the message in MSG format and open in Office Outlook MapiMessage msg = MapiMessage.FromMailMessage(message); msg.SetMessageFlags(MapiMessageFlags.MSGFLAG_UNSENT); msg.Save("Invitation.msg"); Process.Start("Invitation.msg"); //[VB.NET] ' Load the Outlook template (OFT) file in MailMessage's instance Dim message As MailMessage = MailMessage.Load("invitation to meeting.oft", MessageFormat.Msg) ' Set the sender and recipients information Dim senderDisplayName As String = "John" Dim senderEmailAddress As String = "[email protected]" Dim recipientDisplayName As String = "William" Dim recipientEmailAddress As String = "[email protected]" message.Sender = New MailAddress(senderEmailAddress, senderDisplayName) message.To.Add(New MailAddress(recipientEmailAddress, recipientDisplayName)) message.HtmlBody = message.HtmlBody.Replace("DisplayName", "" & recipientDisplayName & "") ' Set the name, location and time in email body Dim meetingLocation As String = "" & "Hall 1, Convention Center, New York, USA" & "" Dim meetingTime As String = "" & "Monday, June 28, 2010" & "" message.HtmlBody = message.HtmlBody.Replace("MeetingPlace", meetingLocation) message.HtmlBody = message.HtmlBody.Replace("MeetingTime", meetingTime) ' Send the email or save as MSG and open in Outlook for further editing Dim client As SmtpClient = New SmtpClient("host", 25, "username", "password") client.Send(message) ' Save the message in MSG format and open in Office Outlook Dim msg As MapiMessage = MapiMessage.FromMailMessage(message) msg.SetMessageFlags(MapiMessageFlags.MSGFLAG_UNSENT) msg.Save("Invitation.msg") Process.Start("Invitation.msg")

Story points are a common name for sizing stories in agile projects. Combined with XpVelocity they provide a technique to aid planning by providing a forecast of when stories can be completed. When estimating work, a common approach is to estimate in terms staff-hours, such as a programmer saying "this will take me two days to do". Many people in the early days of agile, especially those in the ExtremeProgramming community, found that teams struggled to come up with useful estimates using this approach, even when they applied an approach of IdealTime. We found the most effective way to estimate was to size stories relative to each other, and then use past experience to determine how much could be done in an iteration. [1] To determine the points for a story, we compare rough relative sizes. If we are estimating the "fibble the foobar" story, we look for a story of similar size that we've already estimated. We sense it's about the same size as "flipping the synergy bit". Then we look at the story point score for "flipping the synergy bit" and score the "fibble the foobar" the same amount. A team using story points uses a small range of story points to work with. Common examples might be 1,2,4,8 or 1,2,3,5,8 [2]. Often the top number in the series represents "too big" and should be broken down further. [3] Allocating story points should be rapid activity. Discussion should only break out when people have contrasting views on the estimate, in which case its useful to have a discussion as it usually means that something about the story isn't clear. Using a ThrownEstimate is a good technique to move things along quickly. To form a plan with time, you use XpVelocity. Some teams don't like using story points, preferring instead to use StoryCounting. I don't have a preference between the two - both seem to work equally well so it's up to the team to try out and go with whichever suits them best. Further Reading The ThoughtWorks ebook on estimation provides includes a good Q&A on story points. Kent and I discussed them in more depth in the tasteful green book. Most books that talk about planning and estimation in an agile context discuss story points in more detail. Notes 1: "Story Points" is the most common name that I hear these days, but various terms have been used over the years, often with whimsical names that emphasized their arbitrary nature. I particularly like Joseph Pelrine's gummi bears and Josh Kerievsky's: NUTs (Nebulous Units of Time). 2: This is a Fibonacci sequence 3: Using the top number as too big is saying that a story sized at '8' really means '8 or more'. If you do this beware of using this top number when making forecasts of things like completion time, since '8' can turn into all sorts of numbers when it finally gets broken down. It's usually better to explicitly say its too big to be estimated rather than use a false marker number.

Fake system clock is a design pattern addressing testability issues of programs heavily relying on system time. If business logic flow depends on current system time, testing various flows becomes cumbersome or even impossible. Examples of such problematic scenarios include: certain business flow runs only (or is ignored) during weekends some logic is triggered only after an hour since some other event when two events occur at the exact same time (typically 1 ms precision), something should happen … Each scenario above poses unique set of challenges. Taken literally our unit tests would have to run only on specific day (1) or sleep for an hour to observe some behaviour. Scenario (3) might even be impossible to test under some circumstances since system clock can tick 1 millisecond at any time, thus making test unreliable. Fake system clock addresses these issues by abstracting system time over simple interface. Essentially you never call new Date(), new GregorianCalendar() or System.currentTimeMillis() but always rely on this: import org.joda.time.{DateTime, Instant} trait Clock { def now(): Instant def dateNow(): DateTime } As you can see I am depending on Joda Time library. Since we are already in the Scala land, one might consider scala-timeor nscala-time wrappers. Moreover the abstract name Clock is not a coincidence. It’s short and descriptive, but more importantly it mimics java.time.Clock class from Java 8 - that happens to address the same problem discussed here at the JDK level! But since Java 8 is still not here, let’s stay with our sweet and small abstraction. The standard implementation that you would normally use simply delegates to system time: import org.joda.time.{Instant, DateTime} object SystemClock extends Clock { def now() = Instant.now() def dateNow() = DateTime.now() } For the purposes of unit testing we will develop other implementations, but first let’s focus on usage scenarios. In a typical Spring/JavaEE applications fake system clock can be turned into a dependency that the container can easily inject. This makes dependence on system time explicit and manageable, especially in tests: @Controller class FooController @Autowired() (fooService: FooService, clock: Clock) { def postFoo(name: String) = fooService store new Foo(name, clock) } Here I am using Spring constructor injection asking the container to provide some Clock implementation. Of course in this case SystemClock is marked as @Service. In unit tests I can pass fake implementation and in integration tests I can place another, @Primary bean in the context, shadowing the SystemClock. This works great, but becomes painful for certain types of objects, namely entity/DTO beans and utility (static) classes. These are typically not managed by Spring so it can’t inject Clock bean to them. This forces us to pass Clock manually from the last “managed” layer: class Foo(fooName: String, clock: Clock) { val name = fooName val time = clock.dateNow() } similarly: object TimeUtil { def firstFridayOfNextMonth(clock: Clock) = //... } It’s not bad from design perspective. Both Foo constructor and firstFridayOfNextMonth() method do rely on system time so let’s make it explicit. On the other hand Clock dependency must be dragged, sometimes through many layers, just so that it can be used in one single method somewhere. Again, this is not bad per se. If your high level method has Clockparameter you know from the beginning that it relies on current time. But still is seems like a lot of boilerplate and overhead for little gain. Luckily Scala can help us here with: implicit parameters Let us refactor our solution a little bit so that Clock is an implicit parameter: @Controller class FooController(fooService: FooService) { def postFoo(name: String)(implicit clock: Clock) = fooService store new Foo(name) } @Service class FooService(fooRepository: FooRepository) { def store(foo: Foo)(implicit clock: Clock) = fooRepository storeInFuture foo } @Repository class FooRepository { def storeInFuture(foo: Foo)(implicit clock: Clock) = { val friday = TimeUtil.firstFridayOfNextMonth() //... } } object TimeUtil { def firstFridayOfNextMonth()(implicit clock: Clock) = //... } Notice how we call fooRepository storeInFuture foo ignoring second clock parameter. However this alone is not enough. We still have to provide some Clock instance as second parameter, otherwise compilation error strikes: could not find implicit value for parameter clock: com.blogspot.nurkiewicz.foo.Clock controller.postFoo("Abc") ^ not enough arguments for method postFoo: (implicit clock: com.blogspot.nurkiewicz.foo.Clock)Unit. Unspecified value parameter clock. controller.postFoo("Abc") ^ The compiler tried to find implicit value for Clock parameter but failed. However we are really close, the simplest solution is to use package object: package com.blogspot.nurkiewicz.foo package object foo { implicit val clock = SystemClock } Where SystemClock was defined earlier. Here is what happens: every time I call a function with implicit clock: Clock parameter inside com.blogspot.nurkiewicz.foo package, the compiler will discover foo.clock implicit variable and pass it transparently. In other words the following code snippets are equivalent but the second one provides explicit Clock, thus ignoring implicits: TimeUtil.firstFridayOfNextMonth() TimeUtil.firstFridayOfNextMonth()(SystemClock) also equivalent (first form is turned into the second by the compiler): fooService.store(foo) fooService.store(foo)(SystemClock) Interestingly in the bytecode level, implicit parameters aren’t any different from normal parameters so if you want to call such method from Java, passing Clock instance is mandatory and explicit. implicit clock parameter seems to work quite well. It hides ubiquitous dependency while still giving possibility to override it. For example in: fooService.store(foo) fooService.store(foo)(SystemClock) Tests The whole point of abstracting system time was to enable unit testing by gaining full control over time flow. Let us begin with a simple fake system clock implementation that always returns the same, specified time: class FakeClock(fixed: DateTime) extends Clock { def now() = fixed.toInstant def dateNow() = fixed } Of course you are free to put any logic here: advancing time by arbitrary value, speeding it up, etc. You get the idea. Now remember, the reason for implicit parameter was to hide Clock from normal production code while still being able to supply alternative implementation. There are two approaches: either pass FakeClock explicitly in tests: val fakeClock = new FakeClock( new DateTime(2013, 7, 15, 0, 0, DateTimeZone.UTC)) controller.postFoo("Abc")(fakeClock) or make it implicit but more specific to the compiler resolution mechanism: implicit val fakeClock = new FakeClock( new DateTime(2013, 7, 15, 0, 0, DateTimeZone.UTC)) controller.postFoo("Abc") The latter approach is easier to maintain as you don’t have to remember about passing fakeClock to method under test all the time. Of course fakeClock can be defined more globally as a field or even inside test package object. No matter which technique of providing fakeClock we choose, it will be used throughout all calls to service, repository and utilities. The moment we given explicit value to this parameter, implicit parameter resolution is ignored. Problems and summary Solution above to testing systems heavily dependant on time is not free from issues on its own. First of all the implicitClock parameter must be propagated throughout all the layers up to the client code. Notice that Clock is only needed in repository/utility layer while we had to drag it up to the controller layer. It’s not a big deal since the compiler will fill it in for us, but sooner or later most of our methods will include this extra parameter. Also Java and frameworks working on top of our code are not aware of Scala implicit resolution happening at compile time. Therefore e.g. our Spring MVC controller will not work as Spring is not aware of SystemClock implicit variable. It can be worked around though with WebArgumentResolver. Fake system clock pattern in general works only when used consistently. If you have even one place when real time is used directly as opposed to Clock abstraction, good luck in finding test failure reason. This applies equally to libraries and SQL queries. Thus if you are designing a library relying on current time, consider providing pluggable Clock abstraction so that client code can supply custom implementation like FakeClock. In SQL, on the other hand, do not rely on functions likeNOW() but always explicitly provide dates from your code (and thus from custom Clock).

I previously wrote a blog post describing how I’ve been trying to learn more about network sockets in which I created some server sockets and connected to them using netcat. The next step was to do the same thing in Java and I started out by writing a server socket which echoed any messages sent by the client: public class EchoServer { public static void main(String[] args) throws IOException { int port = 4444; ServerSocket serverSocket = new ServerSocket(port, 50, InetAddress.getByAddress(new byte[] {0x7f,0x00,0x00,0x01})); System.err.println("Started server on port " + port); while (true) { Socket clientSocket = serverSocket.accept(); System.err.println("Accepted connection from client: " + clientSocket.getRemoteSocketAddress() ); In in = new In (clientSocket); Out out = new Out(clientSocket); String s; while ((s = in.readLine()) != null) { out.println(s); } System.err.println("Closing connection with client: " + clientSocket.getInetAddress()); out.close(); in.close(); clientSocket.close(); } } } public final class In { private Scanner scanner; public In(java.net.Socket socket) { try { InputStream is = socket.getInputStream(); scanner = new Scanner(new BufferedInputStream(is), "UTF-8"); } catch (IOException ioe) { System.err.println("Could not open " + socket); } } public String readLine() { String line; try { line = scanner.nextLine(); } catch (Exception e) { line = null; } return line; } public void close() { scanner.close(); } } public class Out { private PrintWriter out; public Out(Socket socket) { try { out = new PrintWriter(socket.getOutputStream(), true); } catch (IOException ioe) { ioe.printStackTrace(); } } public void close() { out.close(); } public void println(Object x) { out.println(x); out.flush(); } } I ran the main method of the class and this creates a server socket on port 4444 listening on the 127.0.0.1 interface and we can connect to it using netcat like so: $ nc -v 127.0.0.1 4444 Connection to 127.0.0.1 4444 port [tcp/krb524] succeeded! hello hello The output in my IntelliJ console looked like this: Started server on port 4444 Accepted connection from client: /127.0.0.1:63222 Closing connection with client: /127.0.0.1 Using netcat is fine but what I actually wanted to do was write some test code which would check that I’d made sure the server socket on port 4444 was accessible via all interfaces i.e. bound to 0.0.0.0. There are actually some quite nice classes in Java which make this very easy to do and wiring those together I ended up with the following client code: public static void main(String[] args) throws IOException { Enumeration nets = NetworkInterface.getNetworkInterfaces(); for (NetworkInterface networkInterface : Collections.list(nets)) { for (InetAddress inetAddress : Collections.list(networkInterface.getInetAddresses())) { Socket socket = null; try { socket = new Socket(inetAddress, 4444); System.out.println(String.format("Connected using %s [%s]", networkInterface.getDisplayName(), inetAddress)); } catch (ConnectException ex) { System.out.println(String.format("Failed to connect using %s [%s]", networkInterface.getDisplayName(), inetAddress)); } finally { if (socket != null) { socket.close(); } } } } } } If we run the main method of that class we’ll see the following output (on my machine at least!): Failed to connect using en0 [/fe80:0:0:0:9afe:94ff:fe4f:ee50%4] Failed to connect using en0 [/192.168.1.89] Failed to connect using lo0 [/0:0:0:0:0:0:0:1] Failed to connect using lo0 [/fe80:0:0:0:0:0:0:1%1] Connected using lo0 [/127.0.0.1] Interestingly we can’t even connect via the loopback interface using IPv6 which is perhaps not that surprising in retrospect given we bound using an IPv4 address. If we tweak the second line of EchoServer from: ServerSocket serverSocket = new ServerSocket(port, 50, InetAddress.getByAddress(new byte[] {0x7f,0x00,0x00,0x01})); to ServerSocket serverSocket = new ServerSocket(port, 50, InetAddress.getByAddress(new byte[] {0x00,0x00,0x00,0x00})); And restart the server before re-running the client we can now connect through all interfaces: Connected using en0 [/fe80:0:0:0:9afe:94ff:fe4f:ee50%4] Connected using en0 [/192.168.1.89] Connected using lo0 [/0:0:0:0:0:0:0:1] Connected using lo0 [/fe80:0:0:0:0:0:0:1%1] Connected using lo0 [/127.0.0.1] We can then wrap the EchoClient code into our testing framework to assert that we can connect via all the interfaces.

What is the hardest thing about writing code? There are many common answers to this question: Learning a new technology Naming things Testing your code Debugging Fixing bugs Making software maintainable The list goes on and on. But as I reflect back on my programming career, and I’ve conversed with many new programmers that are learning the craft, I’ve found the single hardest thing about programming is learning the problem domain. A familiar problem In a good portion of my Pluralsight courses I show the viewer how to build the “Protein Tracker” application. I am often asked why I keep demonstrating how to build the same simple application over and over again in each of my courses. The answer is “familiarity.” When I first started using the Protein Tracker example in my Android course, I was just looking for a simple example of an application that could be easily understood and implemented. The idea behind the Protein Tracker application is that it allows a user to set a goal for the amount of protein to consume in a day. The user can add protein amounts which are added to a total protein count that is tracked for that user. Very simple functionality, easily explainable, but most importantly, easily understood. What I found with this simple application was that because it was so easy to understand, the focus was taken off of the problem domain and put instead on the technology. Not only this, but as I reused this same exact application for teaching a variety of different technologies, it served as a reference problem domain that didn’t have to be re-learned, and provided a way to compare and contrast different technologies—I was getting this teaching mechanism for free if a viewer had already watched one of my other Pluralsight courses. By creating a familiar problem domain, I found that both the tasks of me teaching a new technology and the viewer learning that technology were much easier, because it is very difficult to learn more than one thing at once. So what am I trying to say here? Simply that by taking away the problem domain, or making it so trivial that it is easily understood, I am able to make both teaching and learning easier. Why problem domains are hard Have you ever tried to put together a jigsaw puzzle that didn’t have any picture on it? How about one like this one, that has a very similar pattern repeated on it and is double-sided? The reason why puzzles like this one are so hard, is because you can’t really see what you are trying to build very clearly. Normally when you put together a jigsaw puzzle you follow steps that might look something like this: Figure out what the major components of the picture are Sort the pieces by color or component Put together all the border pieces Put together each component of the picture from the piles you created This all breaks down when you don’t have a picture with clear components that you can identify. The same thing happens when writing code. Writing code is a lot like putting together a jigsaw puzzle. We put together code with the purpose of building components that we have taken out of the “bigger picture” of the problem domain. The big issue is that many problem domains are like a puzzle with a blurry picture or no picture at all. The real world is a messy place. Many of the problem domains we face as programmers are difficult to understand and look completely different depending on your viewpoint. As programmers, we also are often not given complete information about the problem domain, so we don’t even have the information we need to understand it. Just try and read the famous Domain Driven Design book and you’ll quickly see how complicated and difficult problem domains can be. (Great book by the way, although you may have to read it twice or three times—I certainly did.) Programming is easy if you understand the problem domain A long time ago, I worked for Hewlett Packard writing software for multi-function printers. Most of the work at the time was basic waterfall development. There wasn’t much Agile happening there—at least at the time I was there. There was however something really interesting about the waterfall approach and the extreme amount of specification that was done before anything was built—it was very easy to write the code for a feature. I remember writing a tab control for the user interface of a printer and having the complete pixel perfect specs handed to me before I began to write any code. I was also given all the possible use cases and told exactly how it should function and what it should do under just about every circumstance. Guess how easy it was to write the code to produce this tab control? Super easy. As much as I frown upon this approach for software development today, there is something interesting to think about here. I was essentially given the entire problem domain in the form of a spec that was clear and unambiguous. I was easily able to learn that problem domain and because of it, I was able to write the code very easily as well. Perhaps you have had a similar experience, not necessarily working on a waterfall project where you were given the spec, but perhaps on an Agile project where you took the time to clearly understand the problem domain before writing any code. I’ve spent days trying to implement a feature only to finally go back and talk to a product owner and hash out completely how something should work and why it should work in a particular way, only to go back to my desk and crank out the code in a matter of hours. The more and more I write code, the more I learn that understanding the problem is the most critical piece to the equation. It is very difficult to solve a problem before you know the question. It’s like buzzing in on Jeopardy before you hear the clue and shouting out random questions. What can you do about it? If understanding the problem domain is the hardest part of programming and you want to make programming easier, you can do one of two things: Make the problem domain easier Get better at understanding the problem domain You can often make the problem domain easier by cutting out cases and narrowing your focus to a particular part of the problem. What I mean by this is that it is often beneficial to take a part of the problem and fully understand that part before expanding the problem domain. Games are really good at this. Look at most games today and you’ll find that you start with a very small problem domain. The first level is usually a tutorial that has a basic set of things you can do so that you don’t get overwhelmed. But, as you advance through the levels, you usually find they get harder and introduce new concepts that build gradually on what you know, until you understand a pretty large problem domain. (Starcraft is really good at this.) The other choice is to become better at understanding problem domains. As developers, we tend to think that sitting down and talking to customers or business people who know about the problem domain is a waste of time. It is easy to fall into the trap of thinking you understand enough of the problem to get started coding it. Best to resist the temptation to “not waste anymore time talking” and make sure you understand a problem inside and out before you try and solve it with code. It is much more expensive and time consuming to do things over than it is to do them right the first time. I learn this lesson the hard way time and time again. Quick update on my new product I’m still not ready to unveil exactly what I am building, but I do have an active mailing list where you can sign up to find out when I release the product I’m working on to help developers get better career opportunities and market themselves. So many developers don’t realize how much of an impact marketing themselves and branding can have on their opportunities. I’m hoping to help developers learn not only how valuable marketing and branding is, but how to do it most effectively. If you are interested, please sign up. (I won’t spam you.)

There are quite a few configuration libraries available in Java, such as this one available from Apache Commons, and they usually follow a very similar pattern: they parse a variety of configuration files and in the end, give you a Property or Map like structure where you can query your values: Double double = config.getDouble("number"); Integer integer = config.getInteger("number"); I have always been unsatisfied with this approach for a couple of reasons: A lot of boiler plate to retrieve these parameters. Having to share the whole configuration object even if I only need one parameter from it. It’s very easy to misspell a property and received incorrect values. A while ago, I was reading the Guice documentation and I came across a paragraph that made me realize that maybe, we could do better. Here is the relevant excerpt: Guice supports binding annotations that have attribute values. In the rare case that you need such an annotation: Create the annotation @interface. Create a class that implements the annotation interface. Follow the guidelines for equals() and hashCode() specified in the Annotation Javadoc. Pass an instance of this to the annotatedWith() binding clause. I thought that using this technique might be exactly what I needed to create a smarter configuration framework, even though I had different plans than using this trick with the annotatedWith method, as suggested by this paragraph. The relevance of this snippet will become clear later, so let’s start with the goals. Objectives I want to: Be able to inject individual configuration values anyhere in my code base and I want this to be type safe. No @Named or other string-based lookup. Have a canonical list of all the properties available to the application, with their full type, default value, documentation and leaving the door open for improvements (e.g. is this option mandatory or optional, detecting when some properties are not used anywhere, deprecation, aliasing, etc…). I don’t care much about the front end: how these properties get gathered is not relevant to this framework, they can come from XML, JSON, the network, a database, and they can have arbitrarily complex resolution and overriding rules, let’s save this for a future post. The input of this framework is a Map of properties and I take it from there. By the time we’re done, we will be able to do something like this: # Some property file host=foo.com port=1234 Using these configuration values in your code: public class A { @Inject @Prop(Property.HOST) private String host; @Inject @Prop(Property.PORT) private Integer port; // ... } Implementation The definition of the Prop annotation is trivial: @Retention(RUNTIME) @Target({ ElementType.FIELD, ElementType.PARAMETER }) @BindingAnnotation public @interface Prop { Property value(); } Property is an enum that captures all the information necessary for all your properties. In our case: public enum Property { HOST("host", "The host name", new TypeLiteral() {}, "foo.com"), PORT("port", "The port", new TypeLiteral() {}, 1234); } This enum contains the string name of the property, a description, its default value and its type. Note that this type is a TypeLiteral, so we can even offer properties that have generic types that would otherwise be erased, a trick that comes in handy to inject caches or other generic collections. Obviously, you can have additional parameters as you see fit (e.g. “boolean deprecated“). The next step is to tie all the properties that we parsed as input — we’ll use a Map called "allProps"— into our module so that Guice knows how to inject them. In order to do this, we iterate all these properties and bind them to their own provider. Because we are using typed names, note the use of Key.get from the Guice API, which lets us specifically target each property with a specific annotation: for (Property prop : Property.values()) { Object value = PropertyConverters.getValue(prop.getType(), prop, allProps.asMap()); binder.bind(Key.get(prop.getType(), new PropImpl(prop))) .toProvider(new PropertyProvider(prop, value)); } There are three classes in this piece of code that I haven’t explained yet. The first one isPropertyConverters, which simply reads the string version of the property and converts it to a Java type. The second one is PropertyProvider, a trivial Guice provider: public class PropertyProvider implements Provider { private final T value; private final Property property; public PropertyProvider(Property property, T value) { this.property = property; this.value = value; } @Override public T get() { return value; } } PropImpl is more tricky and also the one thing that has always prevented me from implementing such a framework, until I came across this obscure tidbit of the Guice documentation quoted above. In order to understand the necessity of its existence, we need to understand how Guice’s Key.get() works. Guice uses this class to translate a type into a unique key that it can use to inject the correct value. The important part here is to notice that not only does this method work with both Class andTypeLiteral (which we are using), but it can also be given a specific annotation. This annotation can be @Named, which I’m not a big fan of because it’s a string, so susceptible to typos, or a real annotation, which is what we want. However, annotations are special beasts in Java and you can’t get an instance of them just like that. This is where the trick mentioned at the top of this article comes into play: Java actually allows you to implement an annotation with a regular class. The implementation turns out to be fairly trivial, the difficulty was realizing that this was possible at all. Now that we have all this in place, let’s back track and dissect how the magic happens: @Inject @Prop(Property.HOST) private String host; When Guice encounters this injection point, it looks into its binders and it finds multiple bindings forStrings. However, because they have all been bound with a Key, the key is actually a pair: (String, a Prop). In this case, it will look up the pair String, Property.HOST and it will find a provider there. This provider was instantiated with the value found in the property file, so it knows what value to return. Generalizing Once I had the basic logic in place, I wondered if I could turn this mini framework into a library so that others could use it. The only missing piece would be to allow the specification of a more general Propannotation. In the example above, this annotation has a value of type Property, which is specific to my application: @Retention(RUNTIME) @Target({ ElementType.FIELD, ElementType.PARAMETER }) @BindingAnnotation public @interface Prop { Property value(); } In order to make this more general, I need to make this attribute return an enum instead of my own: @Retention(RUNTIME) @Target({ ElementType.FIELD, ElementType.PARAMETER }) @BindingAnnotation public @interface Prop { Enum value(); } Unfortunately, this is not legal Java, because according to the JLS section 8.9, Enum and its generic variants are not enum types, something that Josh Bloch confirmed, to my consternation. Therefore, this cannot be turned into a library, so if you are interested in using it in your project, you will have to copy the source and make a few modifications to adjust it to your needs, starting by havingProp#value have the type of the enum that captures your configuration. You can find a small proof of concept here, which I hope you’ll find useful. Note: this is a copy of the article I posted on our work blog.

A question every customer asks me is: How can you manage and monitor integration routes implemented with Apache Camel (http://camel.apache.org/) and / or Talend ESB (http://en.talend.com/products/esb) (which is based on Apache Camel and also available as open source version)? This blog post will show different alternatives to answer this question. The good news first: As Apache Camel and Talend ESB are based on open standards, you can use your own frameworks and tools if tooling of the product is not sufficient. So, I will not talk just about features of Apache Camel or Talend ESB, but also about additional options. Except for Talend Administration Center (user interface on top of Talend ESB’s service monitoring and clustering features), all tools are open source. Talend Administration Center is only available in Talend’s enterprise editions. Management of Apache Camel and Talend ESB Let’s begin with management of Apache Camel and Talend ESB routes. I define management as “controlling server instances, services and integration routes”. Besides Talend Administration Center, a Talend-specific web application, several other options exist. I will show jconsole and hawtio. However, as Apache Camel and Talend ESB are standards-based, you can use any other management tool, which supports the Java Management Extensions (JMX) standard, e.g. Hyperic, Nagios or IBM Tivoli. Talend Administration Center Talend Administration Center (http://www.talendforge.org/tutorials/tutorial.php?idTuto=54) is a web application to configure and monitor different server instances. Besides, you can install, start and stop Camel routes and SOAP / REST web services. Provisioning (i.e. deploying and managing routes and services on different server instances) can be automated easily because Talend ESB container supports Apache Karaf Cellar, an OSGi addon which is built exactly for this reason. Therefore, you can create so-called “virtual servers” within Talend Administration Center. Service locator is another feature, which is based on Apache Zookeeper to implement load balancing and high availability under the hood. Administrators can monitor the state of server instances, Camel routes and REST / SOAP web services within Talend Administration Center. Since Talend ESB 5.3, Talend Administration Center contains other important enterprise features, especially management of Talend’s new service registry (based on Apache Jackrabbit) for service discovery and policy enforcement, and its new identity management (based on Apache Syncope) for authorization. Let’s now talk about two other alternatives for managing Camel and Talend ESB via JMX standard interface. jconsole jconsole is a graphical monitoring tool to monitor Java Virtual Machine (JVM) and java applications both on a local or remote machine. jconsole uses underlying features of JVM to provide information on performance and resource consumption of applications running on the Java platform using JMX technology. jconsole comes as part of Java Development Kit (JDK) and the graphical console can be started using “jconsole” command. Apache Camel and Talend ESB offer several JMX interfaces to monitor and manage integration routes, endpoints, error handlers, etc. jconsole has a very simple user interface, but is sufficient for some projects. hawtio hawtio (http://hawt.io) is a highly modular single-page JavaScript application that fetches JMX metric data using Jolokia, a JMX-HTTP bridge. hawtio is a relatively new tool “on the market”. Nevertheless, it already contains plugins for several different technologies and frameworks such as Apache Camel (integration framework), Apache Karaf (OSGi container), Apache ActiveMQ (JMS messaging) and Apache Tomcat (Java EE web container). As Talend ESB uses respectively supports all of these technologies and frameworks, hawtio is a perfect match, too! hawtio is not “just another jconsole”, but offers some very cool additional features, for example a graphical presentation of Camel routes including real-time monitoring. hawtio was created by James Strachan (whom else?!) who is the creator of Apache Camel and some other great frameworks. Therefore, at the moment, most contributions come from JBoss (where James works after JBoss’ acquisition of FuseSource last year). In the future, I hope (and expect) to see other companies supporting this great open source project, too. Monitoring of Apache Camel and Talend ESB Let’s continue with some monitoring alternatives for Apache Camel and Talend ESB. I define monitoring as “searching and (visually) analyzing state of integration routes and services at runtime”. I will show you Talend’s Service Activity Monitoring for SOAP and REST web services. Besides, you will learn about logstash and Kibana, two awesome open source tools (based on ElasticSearch) for analysis and visualization of any (distributed) log files. Service Activity Monitoring (Part of Talend Administration Center) Service Activity Monitoring (SAM, http://www.liquid-reality.de/display/liquid/Talend+ESB+Service+Activity+Monitoring) is part of Talend Administration Center and allows for logging and monitoring SOAP and REST web service calls. For example, SAM could be used for collecting usage statistics and fault monitoring. SAM is a part of Talend’s OSGi container and can be installed via a single command. It can be activated for SOAP and REST web service providers and consumers within Talend Studio via a single click. So, without any additional efforts, you can monitor your SOAP and REST web services. You can sort by several different parameters and step into details within Talend Administration Center, i.e. you can look at every incoming request message and outgoing response message. logstash logstash (http://logstash.net) is a tool for managing events and logs. You can use it to collect different (distributed) logs, parse them, and store them for later use. Speaking of searching, logstash comes with a simple, but fine web interface for searching and drilling into all of your logs. It uses ElasticSearch under the hood. So, you can easily query through all your logs for specific errors or business analytics (e.g. searching for all lines matching an unique order id). I never used logstash before. I really love it now! However, documentation is not perfect for getting started. So, two hints for newbies: 1) logstash uses ElasticSearch under the hood. However, you do not have to install it separately. It is included and can be used as embedded server. 2) I thought logstash can process every log file by default. However, your log files have to contain a specific structure or you have to configure logstash to know your custom log structure. Especially, if logstash does not know the correct structure of your timestamp, most features will not work. This was not obvious to me. It took me some time to find out that nothing is happening due to wrong log structures (no errors or exceptions came up, just nothing happened at all). Apache Camel and Talend ESB do not offer support for logstash implicitly. One solution is to use Enterprise Integration Patterns (http://www.eaipatterns.com) such as Wire Tap or Message Store to implement logging. Afterwards, logstash can access and analyse these logs. logstash is really awesome, but its real power can be reached by combining it with Kibana. So let’s go to the next section. Kibana Kibana (http://three.kibana.org ) is a browser based analytics and search interface to logstash and other timestamped data sets stored in ElasticSearch. Kibana strives to be easy to get started with, while also being flexible and powerful, just like logstash and ElasticSearch. Main difference to logstash is a much more powerful HTML5 based web interface. You can use multiple concurrent search inputs highlight to drill down bar charts create line charts, stacked, unstacked, filled or unfilled, with or without points create Pie and donut charts that compare top terms or the results of multiple queries create custom dashboards with multiple charts and much more… Again, some hints for newbies (as documentation for installation and usage is very, very short without many important details, unfortunately): 1) Kibana uses logstash and ElasticSearch under the hood. Both are not included, but must be installed separately. As logstash contains an embedded ElasticSearch server, you can get started quickly by just installing logstash in advance. 2) Kibana installation is tricky! If you google, go to Kibana website and navigate to its installation guide (http://kibana.org/intro.html), you will have to install it via Ruby. As always on my Mac, I had some trouble installing Ruby software (version and compiler problems, etc.). But finally I made it… Just to find out that this is an old version. Accidentally, I came to www.three.kibana.org/intro.html, which is the real up-to-date version. Kibana 3 !!! Here, you do NOT need to install Kibana via Ruby. Just download the web application and deploy it to your web container (e.g. Jetty, Tomcat) or Java EE container (e.g. Glassfish, WebSphere). This is so simple and just works with no further configuration. You can use Kibana to analyse and monitor your data as you do with logstash, i.e. you still have to use Enterprise Integration Patterns to log your data first. Afterwards, you can use Kibana’s web interface, which is much more powerful and comfortable than logstash. Finally, great news for Talend users: Kibana (including ElasticSearch) will be integrated into Talend 5.4 for event logging, complex event processing and visual monitoring. I am really looking forward to this awesome feature! Summary Your mission-critical projects need management and monitoring. Today, this is not just possible with complex and expensive tools of large vendors, but also with lightweight frameworks and products such as Apache Camel or Talend ESB. Management and monitoring of Apache Camel and Talend ESB integration routes and SOAP / REST web services is very easy. Several great alternatives are available. Once again, vendor-independent standards such as JXM show their benefits. You can use tooling which is available implicitly in the framework respectively product, or you can integrate your existing tooling very easily. Content from my blog (www.kai-waehner.de/blog). Feedback and other experiences or alternatives appreciated via comments or directly via @KaiWaehner / [email protected] / LinkedIn / Xing. Best regards, Kai Wähner

As you know, there are many frameworks for mobile app development and a growing number of them are based on HTML5. These next-generation tools help developers create mobile apps for phones and tablets without the steep learning curve associated with native SDKs and other programming languages like Objective-C or Java. For countless developers throughout the world, HTML5 represents the future for cross-platform mobile app development. But the question is why? Why has HTML5 become so popular? The widespread adoption of HTML5 involves the emergence of the bring your own device (BYOD) movement. BYOD means that developers can no longer limit application usage to a single platform because consumers want their apps to run on the devices they use every day. HTML5 allows developers to target multiple devices with a single codebase and deliver experiences that closely emulate native solutions, without writing the same application multiple times, using multiple languages or SDKs. The evolution of modern web browsers means that HTML5 can deliver cross-platform, multi-device solutions that mirror behaviors and experiences of “native” apps to the point that it’s often difficult to distinguish between an app written using native development tools and those using HTML. Multiple platform support, time to market and lower maintenance costs are just a few of the advantages inherent in HTML/JavaScript. It’s advantages don’t stop there. HTML’s ability to mitigate long-term risks associated with emerging technologies such as WinRT, ChromeOS, FirefoxOS, and Tizen is unmatched. Simply said, the only code that will work on all these platforms is HTML/JavaScript. Is there a price? Yes, certainly native app consumes less memory and will have faster or more responsive user experience. But in all cases where a web app would work, you can make a step further and create a mobile web app or even packaged application for the store, for multiple platforms, from single codebase. PhoneJS lets you get started fast. PhoneJS for Your Next Mobile Project PhoneJS is a cross-platform HTML5 mobile app development framework that was built to be versatile, flexible and efficient. PhoneJS is a single page application (SPA) framework, with view management and URL routing. Its layout engine allows you to abstract navigation away from views, so the same app can be rendered differently across different platforms or form factors. PhoneJS includes a rich collection of touch-optimized UI widgets with built-in styles for today’s most popular mobile platforms including iOS, Android, and Windows Phone 8. To better understand the principles of PhoneJS development and how you can create and publish applications in platform stores, let’s take a look at a simple demo app called TipCalculator. This application calculates tip amounts due based on a restaurant bill. The complete source code for this app is available here. The app can be found in the AppStore, Google Play, and Windows Store. PhoneJS Application Layout and Routing TipCalculator is a Single-Page Application (SPA) built with HTML5. The start page is index.html, with standard meta tags and links to CSS and JavaScript resources. It includes a script reference to the JavaScript file index.js, where you’ll find the code that configures PhoneJS app framework logic: TipCalculator.app = new DevExpress.framework.html.HtmlApplication({ namespace: TipCalculator, defaultLayout: "empty" }); Within this section, we must specify the default layout for the app. In this example, we’ll use the simplest option, an empty layout. More advanced layouts are available with full support for interactive navigation styles described in the following images: PhoneJS uses well-established layout methodologies supported by many server-side frameworks, including Ruby on Rails and ASP.NET MVC. Detailed information about Views and Layouts can be found in our online documentation. To configure view routing in our SPA, we must add an additional line of code in index.js: TipCalculator.app.router.register(":view", { view: "home" }); This registers a simple route that retrieves the view name from the URL (from the hash segment of the URL). The home view is used by default. Each view is defined in its own HTML file and is linked into the main application page index.html like this: PhoneJS ViewModel A viewmodel is a representation of data and operations used by the view. Each view has a function with the same base name as the view itself and returns the viewmodel for the view. For the home view, the views/home.js script defines the function home which creates the corresponding viewmodel. TipCalculator.home = function(params) { ... }; Three input parameters are used for the tip calculation algorithm: bill total, the number of people sharing the bill, and a tip percentage. These variables are defined as observables, which will be bound to corresponding UI widgets. Note: Observables functionality is supplied by Knockout.js, an important foundation for viewmodels used in PhoneJS. You can learn more about Knockout.js here. This is the code used in the home function to initialize the variables: var billTotal = ko.observable(), tipPercent = ko.observable(DEFAULT_TIP_PERCENT), splitNum = ko.observable(1); The result of the tip calculation is represented by four values: totalToPay, totalPerPerson, totalTip, tipPerPerson. Each value is a dependent observable (a computed value), which is automatically recalculated when any of the observables used in its definition change. Again, this is standard Knockout.js functionality. var totalTip = ko.computed(...); var tipPerPerson = ko.computed(...); var totalPerPerson = ko.computed(...); var totalToPay = ko.computed(...); For an example of business logic implementation in a viewmodel, let’s take a closer look at the observable totalToPay. The total sum to pay is usually rounded. For this purpose, we have two functions roundUp and roundDown that change the value of roundMode (another observable). These changes cause recalculation of totalToPay, because roundMode is used in the code associated with the totalToPay observable. var totalToPay = ko.computed(function() { var value = totalTip() + billTotalAsNumber(); switch(roundMode()) { case ROUND_DOWN: if(Math.floor(value) >= billTotalAsNumber()) return Math.floor(value); return value; case ROUND_UP: return Math.ceil(value); default: return value; } }); When any input parameter in the view changes, rounding should be disabled to allow the user to view precise values. We subscribe to the changes of the UI-bound observables to achieve this: billTotal.subscribe(function() { roundMode(ROUND_NONE); }); tipPercent.subscribe(function() { roundMode(ROUND_NONE); }); splitNum.subscribe(function() { roundMode(ROUND_NONE); }); The complete viewmodel can be found in home.js. It represents a simple example of a typical viewmodel. Note: In a more complex app, it may be useful to implement a structure that modularizes your viewmodels separate from view implementation files. In other words, a file like home.js need not contain the code to implement the viewmodel and instead call a helper function elsewhere for this purpose. In this walkthrough we’re trying to keep things structurally simple. PhoneJS Views Let’s now turn to the markup of the view located in the view/home.html file. The root div element represents a view with the name ‘home’. Within it is a div containing markup for a placeholder called ‘content’. ... A toolbar is located at the top of the view: dxToolbar is a PhoneJS UI widget. It’s defined in the markup using Knockout.js binding. A fieldset appears below the toolbar. To display a fieldset, we use two special CSS classes understood by PhoneJS: dx-fieldset and dx-field. The fieldset contains a text field for the bill total and two sliders for the tip percentage and the number of diners. Two buttons (dxButton) are displayed below the editors, allowing the user to round the total sum to pay. The remaining view displays fieldsets used for calculated results. Total to pay Total per person Total tip Tip per person This completes the description of the files required to create a simple app using PhoneJS. As you’ve seen, the process is simple, straightforward and intuitive. Start, Debug and Build for Stores Starting and debugging a PhoneJS app is just like any other HTML5 based app. You must deploy the folder containing HTML and JavaScript sources, along with any other required file to your web server. Because there is no server-side component to the architectural model, it doesn’t matter which web server you use as long as it can provide file access through HTTP. Once deployed, you can open the app on a device, in an emulator or a desktop browser by simply navigating app’s start page URL. If you want to view the app as it will appear in a phone or tablet within a desktop browser, you will have to override the UserAgent in the browser. Fortunately, this is easy to do with the developer tools that ship as part of today’s modern browsers: If you prefer not to modify UserAgent settings, you can use the Ripple Emulator to emulate multiple device types. At this point you have a web application that will work in the browser on the mobile device and look like native app. Modern mobile browsers provide access to local storage, location api, camera, so good chances are that your app already has anything it needs. Creating Store Ready Applications Using PhoneJS and PhoneGap But what if you need access to device features that browser does not provide? What if you want an app in the app store, not just a webpage. Then you’ll have to create a hybrid application and de-facto standard for such an app is Apache Cordova aka PhoneGap. PhoneGap project for each platform is a native app project that contains WebView (browser control) and a “bridge” that lets your JavaScript code inside WebView access native functions provided by PhoneGap libraries and plugins. To use it, you need to have SDK for each platform you are targeting, but you don’t need to know details of native development, you just need to put your HTML, CSS, JS files into right places and specify your app’s properties like name, version, icons, splashcreens and so on. To be able to publish your app, you will need to register as developer in the respective development portal. This process is well documented for each store and beyond the scope of this article. After that you’ll be able to receive certificates to sign your app package. The need to have SDK for each platform installed sounds challenging - especially after “write one, run everywhere” promise of HTML5/JS approach. This is a small price to pay for building hybrid application and have everything under control. But still there are several services and products that solves this problem for you. One is Adobe’s online service - PhoneGap Build which allows you to build one app for free (to build more, you’ll need a paid account). If you have all the required platform certificate files, the service can build your app for all supported platforms with a few mouse clicks. You only need to prepare app descriptions, promotional and informational content and icons in order to submit your app to an individual store. For Visual Studio developers, DevExpress offers a product called DevExtreme (it includes PhoneJS), which can build applications for iOS, Android and Windows Phone 8 directly within the Microsoft Visual Studio IDE. To summarize, if you need a web application that looks and feels like native on a mobile device, you need PhoneJS - it contains everything required to build touch-enabled, native-looking web application. If you want to go further and access device features, like the contact list or camera, from JavaScript code, you will need Cordova aka PhoneGap. PhoneGap also lets you compile your web app into a native app package. If you don’t want to install an SDK for each platform you are targeting, you can use the PhoneGapBuild service to build your package. Finally, if you have DevExtreme, you can build packages right inside Visual Studio.

I have to admit that NHibernate provides a really flexible way of handling class inheritance and parent-child relationship.

Overview OpenHFT/Java Lang started as an Apache 2.0 library to provide the low level functionality used by Java Chronicle without the need to persist to a file. This allows serializable and deserialization of data and random access to memory in native space (off heap) It supports writing and reading enumerable types with object pooling. e.g. writing and reading String without creating an object (if it has been pooled). It also supports writing and read primitive types in binary and text without creating any garbage. Small messages can be serialized and deserialized in under a micro-second. Recent additions Java Lang supports a DirectStore which is like a ByteBuffer but can be any size (up to 40 to 48-bit on most systems) It support 64-bit sizes and offset. It support compacted types, and object serialization. It also supports thread safety features such as volatile reads, ordered (lazy) writes, CAS operations and using an int (4 bytes) as a lock in native memory. Testing a native memory lock in Java This test has one lock and a value which is toggled. One thread changes the value from 0 to 1 and the other switches it from 1 to 0. This goes around 20 million times, but has been run for longer final DirectStore store1 = DirectStore.allocate(1L << 12); final int lockCount = 20 * 1000 * 1000; new Thread(new Runnable() { @Override public void run() { manyToggles(store1, lockCount, 1, 0); } }).start(); manyToggles(store1, lockCount, 0, 1); store1.free(); The manyToggles method is more interesting. Note is using the 4 bytes at offset 0 as a lock. You can arrange any number of locks in native space this way. E.g. you might have fixed length records and want to be able to lock them before updating or access them. You can place a lock at the "head" of the record. private void manyToggles(DirectStore store1, int lockCount, int from, int to) { long id = Thread.currentThread().getId(); assertEquals(0, id >>> 24); System.out.println("Thread " + id); DirectBytes slice1 = store1.createSlice(); for (int i = 0; i < lockCount; i++) { assertTrue( slice1.tryLockNanosInt(0L, 10 * 1000 * 1000)); int toggle1 = slice1.readInt(4); if (toggle1 == from) { slice1.writeInt(4L, to); } else { i--; } slice1.unlockInt(0L); } } The size of the DataStore and the offsets within it are long, allowing you to allocate a continuous block of native memory into the many GB, and access it as you required. On my 2.6 GHz i5 laptop I get the following output for this test Contended lock rate was 9,096,824 per second This looks great but under heavy contention, one thread can be staved out. This is more useful for lots of locks and lower contention. Note: if I drop the timeout from 10 ms to 1 ms, it eventually fails meaning sometimes it takes more then 1 ms to get a lock! Conclusion The Java Lang library is taking the step of making it easier to use native memory with the same functionality available on the heap. The language support is not as good, but if you need to store say 128 GB of data you will get a much better GC behavior using off heap memory.

In the last post I talked about our need for some improved testing tools, our choice of Spock as something to spike, and how mocking looks in Spock. As that blog got rather long, I saved the next installment for a separate post. Today I want to look at stubbing. Stubbing Mocking is great for checking outputs - in the example in the last post, we're checking that the process of encoding an array calls the right things on the way out, if you like - that the right stuff gets poked onto the bsonWriter. Stubbing is great for faking your inputs (I don't know why this difference never occurred to me before, but Colin's talk at Devoxx UK made this really clear to me). One of the things we need to do in the compatibility layer of the new driver is to wrap all the new style Exceptions that can be thrown by the new architecture layer and turn them into old-style Exceptions, for backwards compatibility purposes. Sometimes testing the exceptional cases is... challenging. So I opted to do this with Spock. class DBCollectionSpecification extends Specification { private final Mongo mongo = Mock() private final ServerSelectingSession session = Mock() private final DB database = new DB(mongo, 'myDatabase', new DocumentCodec()) @Subject private final DBCollection collection = new DBCollection('collectionName', database, new DocumentCodec()) def setup() { mongo.getSession() >> { session } } def 'should throw com.mongodb.MongoException if rename fails'() { setup: session.execute(_) >> { throw new org.mongodb.MongoException('The error from the new Java layer') } when: collection.rename('newCollectionName'); then: thrown(com.mongodb.MongoException) } } So here we can use a real DB class, but with a mock Mongo that will return us a "mock" Session. It's not actually a mock though, it's more of a stub because we want to tell it how to behave when it's called - in this test, we want to force it to throw an org.mongodb.MongoException whenever execute is called. It doesn't matter to us what get passed in to the execute method (that's what the underscore means on line 16), what matters is that when it gets called it throws the correct type of Exception. Like before, the when: section shows the bit we're actually trying to test. In this case, we want to callrename. Then finally the then: section asserts that we received the correct sort of Exception. It's not enormously clear, although I've kept the full namespace in to try and clarify, but the aim is that anyorg.mongodb.MongoException that gets thrown by the new architecture gets turned into the appropriate com.mongodb.MongoException. We're sort of "lucky" because the old code is in the wrong package structure, and in the new architecture we've got a chance to fix this and put stuff into the right place. Once I'd tracked down all the places Exceptions can escape and started writing these sorts of tests to exercise those code paths, not only did I feel more secure that we wouldn't break backwards compatibility by leaking the wrong Exceptions, but we also found our test coverage went up - and more importantly, in the unhappy paths, which are often harder to test. I mentioned in the last post that we already did some simple stubbing to help us test the data driver. Why not just keep using that approach? Well, these stubs end up looking like this: private static class TestAsyncConnectionFactory implements AsyncConnectionFactory { @Override public AsyncConnection create(final ServerAddress serverAddress) { return new AsyncConnection() { @Override public void sendMessage(final List byteBuffers, final SingleResultCallback callback) { throw new UnsupportedOperationException(); } @Override public void receiveMessage(final ResponseSettings responseSettings, final SingleResultCallback callback) { throw new UnsupportedOperationException(); } @Override public void close() { } @Override public boolean isClosed() { throw new UnsupportedOperationException(); } @Override public ServerAddress getServerAddress() { throw new UnsupportedOperationException(); } }; } } Ick. And you end up extending them so you can just override the method you're interested in (particularly in the case of forcing a method to throw an exception). Most irritatingly to me, these stubs live away from the actual tests, so you can't easily see what the expected behaviour is. In the Spock test, the expected stubbed behaviour is defined on line 16, the call that will provoke it is on line 19 and the code that checks the expectation is on line 22. It's all within even the smallest monitor's window. So stubbing in Spock is painless. Next!