I was recently working a bit with Twitter’s Storm, and it got me wondering, how does it compare to another high-performance, concurrent-data-processing framework, Akka. WHAT’S AKKA AND STORM? Let’s start with a short description of both systems. Storm is a distributed, real-time computation system. On a Storm cluster, you execute topologies, which process streams of tuples (data). Each topology is a graph consisting of spouts (which produce tuples) and bolts (which transform tuples). Storm takes care of cluster communication, fail-over and distributing topologies across cluster nodes. Akka is a toolkit for building distributed, concurrent, fault-tolerant applications. In an Akka application, the basic construct is an actor; actors process messages asynchronously, and each actor instance is guaranteed to be run using at most one thread at a time, making concurrency much easier. Actors can also be deployed remotely. There’s a clustering module coming, which will handle automatic fail-over and distribution of actors across cluster nodes. Both systems scale very well and can handle large amounts of data. But when to use one, and when to use the other? There’s another good blog post on the subject, but I wanted to take the comparison a bit further: let’s see how elementary constructs in Storm compare to elementary constructs in Akka. COMPARING THE BASICS Firstly, the basic unit of data in Storm is a tuple. A tuple can have any number of elements, and each tuple element can be any object, as long as there’s a serializer for it. In Akka, the basic unit is amessage, which can be any object, but it should be serializable as well (for sending it to remote actors). So here the concepts are almost equivalent. Let’s take a look at the basic unit of computation. In Storm, we have components: bolts andsprouts. A bolt can be any piece of code, which does arbitrary processing on the incoming tuples. It can also store some mutable data, e.g. to accumulate results. Moreover, bolts run in a single thread, so unless you start additional threads in your bolts, you don’t have to worry about concurrent access to the bolt’s data. This is very similar to an actor, isn’t it? Hence a Storm bolt/sprout corresponds to an Akka actor. How do these two compare in detail? Actors can receive arbitrary messages; bolts can receive arbitrary tuples. Both are expected to do some processing basing on the data received. Both have internal state, which is private and protected from concurrent thread access. ACTORS & BOLTS: DIFFERENCES One crucial difference is how actors and bolts communicate. An actor can send a message to any other actor, as long as it has the ActorRef (and if not, an actor can be looked up by-name). It can also send back a reply to the sender of the message that is being handled. Storm, on the other hand is one-way. You cannot send back messages; you also can’t send messages to arbitrary bolts. You can also send a tuple to a named channel (stream), which will cause the tuple (message) to be broadcast to all listeners, defined in the topology. (Bolts also ack messages, which is also a form of communication, to the ackers.) In Storm, multiple copies of a bolt’s/sprout’s code can be run in parallel (depending on theparallelism setting). So this corresponds to a set of (potentially remote) actors, with a load-balancer actor in front of them; a concept well-known from Akka’s routing. There are a couple of choices on how tuples are routed to bolt instances in Storm (random, consistent hashing on a field), and this roughly corresponds to the various router options in Akka (round robin, consistent hashing on the message). There’s also a difference in the “weight” of a bolt and an actor. In Akka, it is normal to have lots of actors (up to millions). In Storm, the expected number of bolts is significantly smaller; this isn’t in any case a downside of Storm, but rather a design decision. Also, Akka actors typically share threads, while each bolt instance tends to have a dedicated thread. OTHER FEATURES Storm also has one crucial feature which isn’t implemented in Akka out-of-the-box: guaranteed message delivery. Storm tracks the whole tree of tuples that originate from any tuple produced by a sprout. If all tuples aren’t acknowledged, the tuple will be replayed. Also the cluster management of Storm is more advanced (automatic fail-over, automatic balancing of workers across the cluster; based on Zookeeper); however the upcoming Akka clustering module should address that. Finally, the layout of the communication in Storm – the topology – is static and defined upfront. In Akka, the communication patterns can change over time and can be totally dynamic; actors can send messages to any other actors, or can even send addresses (ActorRefs). So overall, Storm implements a specific range of usages very well, while Akka is more of a general-purpose toolkit. It would be possible to build a Storm-like system on top of Akka, but not the other way round (at least it would be very hard).

The following are some highlights from Dan North‘s excellent, inspiring, and insightful talk Patterns of Effective Delivery at RootConf 2011. North has a unique take on what agile development is, going beyond the established (and rather rigid) views. I really recommend this talk to learn more about effective teams, about North’s “shocking,” beyond-agile experience, and for great ideas on improving your team. The talk challenges the dogma of some widely accepted principles of “right” software development such as TDD, naming, and the evilness of copy/paste. However the challenge is in a positive way: it makes us think in which contexts these principles really help (in many) and when it might be more effective to (temporarily) postpone them. The result is a much more balanced view giving you a better understanding of their value. A lot of it is inspired by the theory (and practice) of Real Options. What are Patterns of Effective Delivery? Patterns – Strategies that work in a particular context – and not in another (too often we forget the context and to consider the context where a strategy doesn’t work / is counter-productive). Beware: a part of the context is the experience of the developer. For inexperienced devs it might be better to just stick to a process and apply TDD all the time instead of trying to guess when they it is appropriate and when it is not. Effective – Optimize for something: Volume of software produced? Time to market? Learning/discovery? Certanity? User experience? Any of these will work. Delivery – Get stuff that is useful out of the door. Software is not important, the utility it provides is. Know why you write the software. Some of the patterns take years to master and require significant investment before you start seeing the benefits. You might need to fail a few times before getting them right. Disclaimer: These are notes that make sense to me. They will likely make only limited or no sense to people that haven’t heard the talk. It would be best to go and listen to it instead. Selected patterns Spike and Stabilize (or throw away): traditionally we decide whether we are writing production-grade code (with high rigor such as TDD) or just a throw-away spike before we start coding – i.e. at the moment when we know the least about it. We should not decide this up front but “exercise the option of investing in the quality” later, based on experience. Start as a spike and if the code proves valuable, stabilize it, refactor, test etc. Evolve the code based on experience (good naming, quality). Defer the commitment to the quality of the code and optimize for learning An example of spike-and-stabilize regarding test naming: take a test originally named blah – you don’t know what it should do yet but you're experimenting. When the code evolves into something meaningful, name it properly, according to that. Ginger Cake – Copy and paste code, rip irrelevant things out until only the important things are left, then write tests around it. You may end up with code that is similar, but not in the ways you expected. If you started with abstracting, it would be the wrong abstraction. The pattern says: “We know and respect DRY but are not slaves to it.” Short Software Half-Life: 1) We don’t care about the software but the utility it gives us. If writing it gives us better ideas, we can delete it and do the better thing. 2) How would you write the code if 1/2 of it – but you don’t know which half – would be gone in a few weeks? The answer is, start simple (see Spike & St.), extract commonalities, improve quality, etc. For code that has already been around for a while and has proven itself useful; Some architecture styles lend themselves better to such quick evolution – such as small, focused services, popularly known as micro services (see slides, esp. p.42+). “Look at the code as it evolves and decide what to invest in.” (The investment includes thinking about the design.) All code is not equal. Create Urgency – To change a paradigm, the way of thinking, people must be desperate and have no more options along with the knowledge of what to do. Apply this pattern when learning something new. Do it on something real, under self-inflicted pressure. For example, you could commit to do an app with a crazy deadline using some new tech. This would give you a sense of urgency, with no more options. It forces you to learn only the parts you really need. Socratic Testing (coaching style) – Don’t tell the team what’s wrong with their code, which is threatening and thus hard to accept. Pair with them on writing a test and to support the test, make “helper” classes etc. that you’d like to see in the production code. If they really are useful, they will spot it and decide to pull them into the production code. Make them the hero. Respond to their questions with another question. Fits In My Head – we need code that we can understand and reason out (big classes, methods, complex models, etc.). Keep the code simple, optimize for understandability, readability, and obviousness. Build Shared Idioms in the team so that the team members would, given the same context, arrive at the same decisions/design. Something should only differ from the usual way of doing it when there is a good reason for it. Thus a difference provides a hint, difference is data. For example, putting all communication over ZeroMQ at only one place through shared memory. This indicates there is some (most likely performance) reason for it. Communication strategies shouldn’t be picked randomly or ad-hoc. TDD – A pattern that, in a particular context, may make you much more effective. Most of you reading this know what TDD is. Bonus: Micro Services Rough notes from James Lewis’s talk Micro Services: Java, the Unix Way (2013) – especially slides 42+: Use web, do not bypass it – REST, JSON; standardized application protocols and message semantics Small with a single responsibilities (does one thing, fits into one’s head, small enough to rewrite and throw away rather than maintain) Containerless and installed as well-behaved Unix services (executable jar with embedded Jetty + rc.d start scripts and config files) Avoid unnecessary coupling; Domains in different bounded contexts should be distinct; It's ok to have duplication with physical separation to enforce it; There will be common code, but it should be library and infrastructure code; Leverage Conway’s Law to support decoupling Provisioned automatically: “The way to manage the complexity of many small applications is declarative provisioning” (including instance count, scaling, load balancing) Status aware and auto-scaling; In-app status pages; monitoring to autoscaling Each service is entirely decoupled from its clients, scalable, testable and deployable individually Decomposition: This technique goes from product to a set of capabilities (e.g. monitoring, reporting, fulfillment, user) and then to each capability being implemented by a set of small apps/services exposing a uniform interface of Atom Collections. The capabilities form the project by interacting via a uniform interface – HTTP (reverse proxies etc.), HATEOS (link relations drive state changes; its an anti-corruption layer that allows the capability to evolve independently of its clients), and standard media types (usable by many types of clients). Explicit tips from the talk: Divide and conquer – Start on the outside and model business capabilities. Use Conway’s Law to structure teams (and enforce decoupling). The Last Responsible Moment – Don’t decide everything at the point when you know the least. Be of the web, not behind the web. If something is important, make it an explicit part of your design (reify) – an exampoe would be an instance of services creating users by posting to /user. They post a user creation request and get response immediately. The user is then created eventually (reminds me of futures). Favor service choreography over orchestration. Use hypermedia controls to decouple services. Some tools used: SimpleWeb/Jetty, Abdera for Atom, Smoothie charts (JS charts for streaming data), Coda Hale’s metrics, Graphite. Ops: Fabric with boto, AWS, Puppet, … . Remember there are NO SILVER BULLETS. This stuff is hard. Versioning, Integration, Testing, Deployment and eventual consistency are hard for people to wrap their heads around. Note: Comoyo.com, powered by a number of ex-googlers and other smart people, does the same thing. So does Netflix, I believe. Related If you liked this, you might also like Dan North's presentations Accelerating Agile: hyper-performing teams without the hype and Patterns of Effective Teams at NDC Oslo 2013.

cdi (context and dependency injection) is a complete and lightweight injection technology designed for java ee environment. special container objects (ejb,entitymanager), primitive data type elements and java class/objects written by you can be easily managed and injected as well through cdi. every defined java class in each application that configured in cdi standard is a candidate to become an injectable cdi object. this default behavior is provided by @default annotation that was installed per each java class secretly. there is an car class which has a vehicle implementation in the above uml diagram. a random int value is produced in sayvelocity() method and there is an output to the console such as “the car is running at the speed of x” in work() method, where x is represents a number produced randomly. @default // optional public class car implements vehicle { public string work() { return "car is working in "+ sayvelocity()+" kmh."; } public int sayvelocity(){ return threadlocalrandom.current().nextint(20, 240) ; } } if the above car class is in an application activated in cdi environment, it becomes a candidate to be an object managed by cdi. so, what is implied by the activation of cdi? first of all, necessary dependencies need to be included in the classpath for the activation of cdi environment. if you are using an application server like glassfish, cdi can be used without any extra definition because of the existence of cdi libraries on the application server. but if your application is a java se application or is running in lightweight containers such as tomcat, jetty; a cdi library must be added to the project. the reference library of cdi technology is the jboss weld archetype. for this reason, if jboss weld dependencies are added to the project such as the following, first phase of the cdi activation is realized. org.jboss.weld.se weld-se 1.1.10.final to activate the cdi environment, a blank cdi configuration file named beans.xml must be in the application as a requirement of the standard. this file must be located on the /meta-inf/beans.xml path for java se applications and /web-inf/beans.xml path for java ee web applications. the existence necessity of this file may sound silly initially, but the existence of beans.xml in the directories specified above can be considered as a permission given to the container in order to activate cdi environment. activation of cdi environment is automatically started in java ee web applications when beans.xml file is encountered in the /web-inf directory. but it is not the same for a java se application, starting a cdi container is the developer’s job. this case can be seen clearly if you pay attention to the following gallery class: public class gallery { @inject // injection point private vehicle vehicle; public static void main(string[] args) { weld weld = new weld(); weldcontainer weldcontainer = weld.initialize(); gallery gallery = weldcontainer.instance().select(gallery.class).get(); string message= gallery.vehicle.work(); system.out.println("> "+message); } } weldcontainer type object reference in gallery class access to a cdi object which represents the initiated container environment and after this point, cdi objects are accessible and can be made injection procedures through weldcontainer object. after this point, it is used by adopting a cdi object that is a gallery class type through the container instance. in here, a programmatic access to the gallery object is provided. programmatic access is required for the startup of java se applications (as in spring), but you can easily access to all cdi objects in the environment also by annotation-based injection method through the obtained gallery cdi object, e.g. [ private @inject vehicle vehicle; ] an output as the following occurs when the gallery class is being run; the real content and sample code can be accessed in http://en.kodcu.com/2013/06/cdi-default-and-inject-annotations/ hope to see you again..

This is a classic DevOps presentation from a former Velocity conference that I thought people could take another look at or see this for the first time. It's definitely worth it. Velocity 09: John Allspaw, "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr" John Allspaw (Flickr/Yahoo!) and Paul Hammond (Flickr), "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr"

We’ve covered the controversy of a DevOps Team on this blog before. DevOps Teams are dangerous in that many organizations realize that their Dev and Ops groups are so far apart, that they need a neutral, expert group that can bring them together. At the same time, there are increasing reports of DevOps teams becoming yet another silo – and one that is often arrogant and disliked. More silos being exactly the opposite of what we are going for, this is a frightening result. The dangers in a DevOps Team seem to be that they will: End up owning a lot of things, and be a silo onto themselves Be over aggressive in dictating how teams should work A little over a week ago at the IBM Innovate conference in Orlando an attendee shared her successful experience creating a DevOps Liaison Team. I am very intrigued by the naming here. When the team is formed with the name and charter to bring the other groups together, it is hard for them to either own systems or dictate. The other pattern that I liked was what WebMD did in their project to select and implement a deployment automation tool. They went to manages in various traditional silos (Dev, QA, Ops, etc) and asked for a techie who: Did real work Had the respect of their peers The manager would delegate authority to compromise to They ended with a team full of skilled engineers who would work together, but then go back to their own teams once the project was done. However, they had formed solid working relationships cross-silo and could become a liaison between their group and others. Both of these approaches seem to form a DevOps Team of sorts, without creating something evil. They also take chisel to walls between groups rather than trying to reorganize radically all at once. I’m encouraged that our industry is beginning to find some healthy patterns for enterprise DevOps adoption. For more on non-evil DevOps teams, check out our recorded webinar: Building a DevOps Team that Isn’t Evil. What about you? Have you had success forming a dedicated team that helps the rest of the organization grok DevOps? Have you failed? Leave your tips in the comments area below.

Andres and I recently found ourselves wanting to delete a remote branch which had the same name as a tag and therefore the normal way of doing that wasn’t worked out as well as we’d hoped. I created a dummy repository to recreate the state we’d got ourselves into: $ echo "mark" > README $ git commit -am "readme" $ echo "for the branch" >> README $ git commit -am "for the branch" $ git checkout -b same Switched to a new branch 'same' $ git push origin same Counting objects: 5, done. Writing objects: 100% (3/3), 263 bytes, done. Total 3 (delta 0), reused 0 (delta 0) To ssh://[email protected]/markhneedham/branch-tag-test.git * [new branch] same -> same $ git checkout master $ echo "for the tag" >> README $ git commit -am "for the tag" $ git tag same $ git push origin refs/tags/same Counting objects: 5, done. Writing objects: 100% (3/3), 266 bytes, done. Total 3 (delta 0), reused 0 (delta 0) To ssh://[email protected]/markhneedham/branch-tag-test.git * [new tag] same -> same We wanted to delete the remote ‘same’ branch and the following command would work if we hadn’t created a tag with the same name. Instead it throws an error: $ git push origin :same error: dst refspec same matches more than one. error: failed to push some refs to 'ssh://[email protected]/markhneedham/branch-tag-test.git' We learnt that what we needed to do was refer to the full path for the branch when trying to delete it remotely: $ git push origin :refs/heads/same To ssh://[email protected]/markhneedham/branch-tag-test.git - [deleted] same To delete the tag we could do the same thing: $ git push origin :refs/tags/same remote: warning: Deleting a non-existent ref. To ssh://[email protected]/markhneedham/branch-tag-test.git - [deleted] same Of course the tag and branch still exist locally: $ ls -alh .git/refs/heads/ total 16 drwxr-xr-x 4 markhneedham wheel 136B 13 Jun 23:09 . drwxr-xr-x 5 markhneedham wheel 170B 13 Jun 22:39 .. -rw-r--r-- 1 markhneedham wheel 41B 13 Jun 23:08 master -rw-r--r-- 1 markhneedham wheel 41B 13 Jun 23:08 same $ ls -alh .git/refs/tags/ total 8 drwxr-xr-x 3 markhneedham wheel 102B 13 Jun 23:08 . drwxr-xr-x 5 markhneedham wheel 170B 13 Jun 22:39 .. -rw-r--r-- 1 markhneedham wheel 41B 13 Jun 23:08 same So we got rid of them as well: $ git checkout master Switched to branch 'master' $ git branch -d same Deleted branch same (was 08ad88c). $ git tag -d same Deleted tag 'same' (was 1187891) And now they are gone: $ ls -alh .git/refs/heads/ total 8 drwxr-xr-x 3 markhneedham wheel 102B 13 Jun 23:16 . drwxr-xr-x 5 markhneedham wheel 170B 13 Jun 22:39 .. -rw-r--r-- 1 markhneedham wheel 41B 13 Jun 23:08 master $ ls -alh .git/refs/tags/ total 0 drwxr-xr-x 2 markhneedham wheel 68B 13 Jun 23:16 . drwxr-xr-x 5 markhneedham wheel 170B 13 Jun 22:39 .. Out of interest we’d ended up with this situation by mistake rather than by design but it was still fun to do a little bit of git digging to figure out how to solve the problem we’d created for ourselves.

In the code I have quite recently came across a really bad piece of code that based on class casting in terms of performing some actions on objects. Of course the code needed to be refactored but sometimes you can't do it / or don't want to do it (and it should be understandable) if first you don't have unit tests of that functionality. In the following post I will show how to test such code, how to refactor it and in fact what I think about such code ;) Let's take a look at the project structure: As presented in the post regarding Mocktio RETURNS_DEEP_STUBS Answer for JAXB yet again we have the JAXB generated classes by the JAXB compiler in thecom.blogspot.toomuchcoding.model package. Let's ommit the discussion over the pom.xml file since it's exactly the same as in the previous post. In the com.blogspot.toomuchcoding.adapter package we have adapters over the JAXB PlayerDetails class that provides access to the Player interface. There is the CommonPlayerAdapter.java package com.blogspot.toomuchcoding.adapter; import com.blogspot.toomuchcoding.model.Player; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:42 */ public class CommonPlayerAdapter implements Player { private final PlayerDetails playerDetails; public CommonPlayerAdapter(PlayerDetails playerDetails){ this.playerDetails = playerDetails; } @Override public void run() { System.out.printf("Run %s. Run!%n", playerDetails.getName()); } public PlayerDetails getPlayerDetails() { return playerDetails; } } DefencePlayerAdapter.java package com.blogspot.toomuchcoding.adapter; import com.blogspot.toomuchcoding.model.DJ; import com.blogspot.toomuchcoding.model.DefensivePlayer; import com.blogspot.toomuchcoding.model.JavaDeveloper; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:42 */ public class DefencePlayerAdapter extends CommonPlayerAdapter implements DefensivePlayer, DJ, JavaDeveloper { public DefencePlayerAdapter(PlayerDetails playerDetails){ super(playerDetails); } @Override public void defend(){ System.out.printf("Defence! %s. Defence!%n", getPlayerDetails().getName()); } @Override public void playSomeMusic() { System.out.println("Oops I did it again...!"); } @Override public void doSomeSeriousCoding() { System.out.println("System.out.println(\"Hello world\");"); } } OffensivePlayerAdapter.java package com.blogspot.toomuchcoding.adapter; import com.blogspot.toomuchcoding.model.OffensivePlayer; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:42 */ public class OffensivePlayerAdapter extends CommonPlayerAdapter implements OffensivePlayer { public OffensivePlayerAdapter(PlayerDetails playerDetails){ super(playerDetails); } @Override public void shoot(){ System.out.printf("%s Shooooot!.%n", getPlayerDetails().getName()); } } Ok, now let's go to the more interesting part. Let us assume that we have a very simple factory of players: PlayerFactoryImpl.java package com.blogspot.toomuchcoding.factory; import com.blogspot.toomuchcoding.adapter.CommonPlayerAdapter; import com.blogspot.toomuchcoding.adapter.DefencePlayerAdapter; import com.blogspot.toomuchcoding.adapter.OffensivePlayerAdapter; import com.blogspot.toomuchcoding.model.Player; import com.blogspot.toomuchcoding.model.PlayerDetails; import com.blogspot.toomuchcoding.model.PositionType; /** * User: mgrzejszczak * Date: 09.06.13 * Time: 15:53 */ public class PlayerFactoryImpl implements PlayerFactory { @Override public Player createPlayer(PositionType positionType) { PlayerDetails player = createCommonPlayer(positionType); switch (positionType){ case ATT: return new OffensivePlayerAdapter(player); case MID: return new OffensivePlayerAdapter(player); case DEF: return new DefencePlayerAdapter(player); case GK: return new DefencePlayerAdapter(player); default: return new CommonPlayerAdapter(player); } } private PlayerDetails createCommonPlayer(PositionType positionType){ PlayerDetails playerDetails = new PlayerDetails(); playerDetails.setPosition(positionType); return playerDetails; } } Ok so we have the factory that builds Players. Let's take a look at the Service that uses the factory: PlayerServiceImpl.java package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.factory.PlayerFactory; import com.blogspot.toomuchcoding.model.*; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:02 */ public class PlayerServiceImpl implements PlayerService { private PlayerFactory playerFactory; @Override public Player playAGameWithAPlayerOfPosition(PositionType positionType) { Player player = playerFactory.createPlayer(positionType); player.run(); performAdditionalActions(player); return player; } private void performAdditionalActions(Player player) { if(player instanceof OffensivePlayer){ OffensivePlayer offensivePlayer = (OffensivePlayer) player; performAdditionalActionsForTheOffensivePlayer(offensivePlayer); }else if(player instanceof DefensivePlayer){ DefensivePlayer defensivePlayer = (DefensivePlayer) player; performAdditionalActionsForTheDefensivePlayer(defensivePlayer); } } private void performAdditionalActionsForTheOffensivePlayer(OffensivePlayer offensivePlayer){ offensivePlayer.shoot(); } private void performAdditionalActionsForTheDefensivePlayer(DefensivePlayer defensivePlayer){ defensivePlayer.defend(); try{ DJ dj = (DJ)defensivePlayer; dj.playSomeMusic(); JavaDeveloper javaDeveloper = (JavaDeveloper)defensivePlayer; javaDeveloper.doSomeSeriousCoding(); }catch(ClassCastException exception){ System.err.println("Sorry, I can't do more than just play football..."); } } public PlayerFactory getPlayerFactory() { return playerFactory; } public void setPlayerFactory(PlayerFactory playerFactory) { this.playerFactory = playerFactory; } } Let's admit it... this code is bad. Internally when you look at it (regardless of the fact whether it used instance of operator or not) you feel that it is evil :) As you can see in the code we have some class casts going on... How on earth can we test it? In the majority of testing frameworks you can't do such class casts on mocks since they are built with the CGLIB library and there can be some ClassCastExceptions thrown. You could still not return mocks and real implementations (assuming that those will not perform any ugly stuff in the construction process) and it could actually work but still - this is bad code :P Mockito comes to the rescue (although you shouldn't overuse this feature - in fact if you need to use it please consider refactoring it) with its extraInterfaces feature: extraInterfaces MockSettings extraInterfaces(java.lang.Class... interfaces) Specifies extra interfaces the mock should implement. Might be useful for legacy code or some corner cases. For background, see issue 51 hereThis mysterious feature should be used very occasionally. The object under test should know exactly its collaborators & dependencies. If you happen to use it often than please make sure you are really producing simple, clean & readable code. Examples: Foo foo = mock(Foo.class, withSettings().extraInterfaces(Bar.class, Baz.class)); //now, the mock implements extra interfaces, so following casting is possible: Bar bar = (Bar) foo; Baz baz = (Baz) foo; Parameters:interfaces - extra interfaces the should implement. Returns:settings instance so that you can fluently specify other settings Now let's take a look at the test: PlayerServiceImplTest.java package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.factory.PlayerFactory; import com.blogspot.toomuchcoding.model.*; import org.junit.Test; import org.junit.runner.RunWith; import org.mockito.InjectMocks; import org.mockito.Mock; import org.mockito.invocation.InvocationOnMock; import org.mockito.runners.MockitoJUnitRunner; import org.mockito.stubbing.Answer; import static org.hamcrest.CoreMatchers.is; import static org.junit.Assert.assertThat; import static org.mockito.BDDMockito.*; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:26 */ @RunWith(MockitoJUnitRunner.class) public class PlayerServiceImplTest { @Mock PlayerFactory playerFactory; @InjectMocks PlayerServiceImpl objectUnderTest; @Mock(extraInterfaces = {DJ.class, JavaDeveloper.class}) DefensivePlayer defensivePlayerWithDjAndJavaDevSkills; @Mock DefensivePlayer defensivePlayer; @Mock OffensivePlayer offensivePlayer; @Mock Player commonPlayer; @Test public void shouldReturnOffensivePlayerThatRan() throws Exception { //given given(playerFactory.createPlayer(PositionType.ATT)).willReturn(offensivePlayer); //when Player createdPlayer = objectUnderTest.playAGameWithAPlayerOfPosition(PositionType.ATT); //then assertThat(createdPlayer == offensivePlayer, is(true)); verify(offensivePlayer).run(); } @Test public void shouldReturnDefensivePlayerButHeWontBeADjNorAJavaDev() throws Exception { //given given(playerFactory.createPlayer(PositionType.GK)).willReturn(defensivePlayer); //when Player createdPlayer = objectUnderTest.playAGameWithAPlayerOfPosition(PositionType.GK); //then assertThat(createdPlayer == defensivePlayer, is(true)); verify(defensivePlayer).run(); verify(defensivePlayer).defend(); verifyNoMoreInteractions(defensivePlayer); } @Test public void shouldReturnDefensivePlayerBeingADjAndAJavaDev() throws Exception { //given given(playerFactory.createPlayer(PositionType.GK)).willReturn(defensivePlayerWithDjAndJavaDevSkills); doAnswer(new Answer

Sorry for not having written for some time but I was busy with writing the JBoss Drools Refcard for DZone and I am in the middle of writing a book about Mockito so I don't have too much time left for blogging... Anyway quite recently on my current project I had an interesting situation regarding unit testing with Mockito and JAXB structures. We have very deeply nested JAXB structures generated from schemas that are provided for us which means that we can't change it in anyway. Let's take a look at the project structure: The project structure is pretty simple - there is a Player.xsd schema file that thanks to using the jaxb2-maven-plugin produces the generated JAXB Java classes corresponding to the schema in the target/jaxb/ folder in the appropriate package that is defined in the pom.xml. Speaking of which let's take a look at the pom.xml file. The pom.xml : 4.0.0 com.blogspot.toomuchcoding mockito-deep_stubs 0.0.1-SNAPSHOT UTF-8 1.6 1.6 spring-release http://maven.springframework.org/release maven-us-nuxeo https://maven-us.nuxeo.org/nexus/content/groups/public junit junit 4.10 org.mockito mockito-all 1.9.5 test org.apache.maven.plugins maven-compiler-plugin 2.5.1 org.codehaus.mojo jaxb2-maven-plugin 1.5 xjc xjc com.blogspot.toomuchcoding.model ${project.basedir}/src/main/resources/xsd Apart from the previously defined project dependencies, as mentioned previously in the jaxb2-maven-plugin in the configuration node you can define the packageName value that defines to which package should the JAXB classes be generated basing on the schemaDirectory value where the plugin can find the proper schema files. Speaking of which let's check the Player.xsd schema file (simillar to the one that was present in the Spring JMS automatic message conversion article of mine): As you can see I'm defining some complex types that even though might have no business sense but you can find such examples in the real life :) Let's find out how the method that we would like to test looks like. Here we have the PlayerServiceImpl that implements the PlayerService interface: package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:02 */ public class PlayerServiceImpl implements PlayerService { @Override public boolean isPlayerOfGivenCountry(PlayerDetails playerDetails, String country) { String countryValue = playerDetails.getClubDetails().getCountry().getCountryCode().getCountryCode().value(); return countryValue.equalsIgnoreCase(country); } } We are getting the nested elements from the JAXB generated classes. Although it violates the Law of Demeter it is quite common to call methods of structures because JAXB generated classes are in fact structures so in fact I fully agree with Martin Fowler that it should be called the Suggestion of Demeter. Anyway let's see how you could test the method: @Test public void shouldReturnTrueIfCountryCodeIsTheSame() throws Exception { //given PlayerDetails playerDetails = new PlayerDetails(); ClubDetails clubDetails = new ClubDetails(); CountryDetails countryDetails = new CountryDetails(); CountryCodeDetails countryCodeDetails = new CountryCodeDetails(); playerDetails.setClubDetails(clubDetails); clubDetails.setCountry(countryDetails); countryDetails.setCountryCode(countryCodeDetails); countryCodeDetails.setCountryCode(CountryCodeType.ENG); //when boolean playerOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); //then assertThat(playerOfGivenCountry, is(true)); } The function checks if, once you have the same Country Code, you get a true boolean from the method. The only problem is the amount of sets and instantiations that take place when you want to create the input message. In our projects we have twice as many nested elements so you can only imagine the number of code that we would have to produce to create the input object... So what can be done to improve this code? Mockito comes to the rescue to together with the RETURN_DEEP_STUBS default answer to the Mockito.mock(...) method: @Test public void shouldReturnTrueIfCountryCodeIsTheSameUsingMockitoReturnDeepStubs() throws Exception { //given PlayerDetails playerDetailsMock = mock(PlayerDetails.class, RETURNS_DEEP_STUBS); CountryCodeType countryCodeType = CountryCodeType.ENG; when(playerDetailsMock.getClubDetails().getCountry().getCountryCode().getCountryCode()).thenReturn(countryCodeType); //when boolean playerOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetailsMock, COUNTRY_CODE_ENG); //then assertThat(playerOfGivenCountry, is(true)); } So what happened here is that you use the Mockito.mock(...) method and provide the RETURNS_DEEP_STUBS answer that will create mocks automatically for you. Mind you that Enums can't be mocked that's why you can't write in the Mockito.when(...) functionplayerDetailsMock.getClubDetails().getCountry().getCountryCode().getCountryCode().getValue(). Summing it up you can compare the readability of both tests and see how clearer it is to work with JAXB structures by using Mockito RETURNS_DEEP_STUBS default answer. Naturally sources for this example are available at BitBucket and GitHub.

Serialization is a form of persistence: serialized data survives the process and the RAM where it was created and can be reconstituted inside different processes and machines that live in a different time or place. Sometimes serialization is a poor form of persistence in fact, one that confuses the boundary between the different schemas the data can fit in. However, what I found useful in the last years of development is to institute a strict separation: serialize Value Objects, Entities, and everything that represents the state of the application. Meanwhile, use Dependency Injection over services that are part of a larger object graph and never serialize this second kind of objects. In the discussion that follows, I make the assumption that serialization and deserialization occur on the same machine (e.g. like for web-oriented sessions.) The problem with serialization, which work transparently most of the time, is the need to serialize service objects instead of limiting the procedure to data structures. How can you store such objects? Not options Some options to solve this problems are really not options. Serialization by itself will fail because of the staleness of the references contained in these objects. For example, in PHP trying to serialize a database connections composed by a Repository or DAO object will rightly fail with an exception. Whenever an object represents a resource of the current machine, it cannot usually be serialized except in the case when the only resource involved is RAM. If the resource is disk space or other running processes such as a database daemon, the reconstitution of the object in another place and time will fail and it's best to just stop the developer immediately during storage. Quasi-options Some solutions to the problem try to avoid the staleness problem by serializing objects without their resources, and make them regrab a new version of them on deserialization. In PHP for example, this can be done with the __sleep() and __wakeup() magic methods, called automatically during serialization and deserializaton respectively. This deserialization mechanism introduces a dependency from the serialized Entity to external services: such a dependency is already in place when building the object the first time (passing the XService in the constructor) but it is aggravated when deserializing (depending on a XServiceFactory instead of just an XService). An improvement, from the dependencies point of view, is to reattach collaborators to deserialized objects like you would for other persistence-related tasks. For example, EntityRepository can inject the missing pieces of Entity every time its find() method is called. However, there is still another option, which is the most resilient from the modelling point of view and not only that of dependency management: injecting non-serializable collaborators through the stack. Objects can collaborate even without keeping field references to each other, and injecting dependencies as parameters move the dependency starting point from the server to the client object (which may or may not be desirable). What is most important is that Entities are relieved of having to manage external references in any context, not only that of persistence and in particular serialization. The metaphor for the 3rd option Misko Hevery likes to say: have you ever seen a credit card able to charge itself? If a CreditCard is an Entity in your domain, it would be very strange to keeping a wire attached to your wallet wherever you go. With the first option, you have the card spring a wire when it is taken out of the wallet, like in horror movies. This intelligent cable tries as its best to attach to the nearest Point of Sale (a bad case of bluetooth I think). With Repositories in mind, you're not dealing with automated wires anymore, but you're still attaching cables between cards and fixed devices. In reality, cards collaborate with the PoS in a fast process that does not last more than a few seconds. Actually, sometimes they don't touch it at all, as in all Internet-based purchases. Keeping services around to deal with external dependencies does not mean the API of your Domain Model has to be biased towards service objects: pos.charge(creditCard); // can equivalently be: creditCard.chargeOn(pos); This is a form of Double Dispatch since there are two objects collaborating and you can dispatch (send messages) to both, being polimorphic by substituting both objects. The sequence of calls is: client -> creditCard -> pos The client object still looks at CreditCard as a behaviorally complete object, but it is clear which dependency is necessary to run each use case (CreditCard method). You can persist a CreditCard easily and send it over the wire to caches or databases. When it comes the time to charge, it is the client that has to bring forward a service able to connect to a bank.

"Don't tell people how to do things, tell them what to do and let them surprise you with their results" - General George S. Patton What's the best way to encourage agile teamwork? It's a tricky question, because so much of Scrum and Kanban practice is predicated on the assumption that collaborative behavior will "happen". Empowerment is often presented as the mechanism for achieving this success. If you just press the empowerment button, developers will then choose to self-organize and will go on to deliver sterling results. Patently however, that isn't the case. I'm sure that many of us will have experienced teams that are actually less than the sum of their parts. Technically skilled people can be more focused on stack traces than on individuals and interactions, and may view each other as unwanted complications or impediments. All too often the social graces that underpin effective agile teamwork have to be elicited painfully, like drawing teeth. Whenever I consider this matter, the above quote by Patton often comes into my mind. It isn't the perspicacity of his argument that I find compelling, or even that it was said so long ago. I suppose that these days we have just become more accepting of such observations. No...to me the interesting thing about this quote is that someone of Patton's background and temperament said it. You see, George Smith Patton was arguably the most hard-boiled U.S. General in World War 2. He was spit-and-polish to the core, and an absolute stickler for discipline. Even tiny misdemeanours would incur his wrath. His idea of a touchy-feely management style was to kick people in the pants after slapping them about the chops, and he frequently railed against "malingerers" who he reckoned ought to be court-martialed and shot. We have yet to hear Esther Derby or Johanna Rothman prescribe such remedies for disaffected team members. Perhaps the most politically correct thing we can do is to categorize his beliefs as an alternative viewpoint. Anyway, it's difficult to imagine anyone less likely than Patton to be sympathetic to agile principles, nor anyone more likely to try and micro-manage those they might consider to be their sub-ordinates. It seems we need a deeper insight if we are to explain this unlikely patronage of a central maxim of agile development. I suspect that Patton knew that if a team is to self-organize and deliver value successfully, then discipline will be key. It can't really be about empowerment, because an empowered team can still be sloppy and never cut the mustard. While good management isn't about telling people how to do their jobs, it is about making sure that they understand the rules of best practice and are competent to follow them, preferably with very little oversight. Strangely perhaps, this is a route to freedom rather than constraint. It releases individual initiative. I think that's what Patton was getting at. Who are we to empower others, after all? What gift is that? Where is the transfer of value? How much better it is to instil the best practices that make people more effective, and thereby become more valued themselves. Development Team Membership Now, a development team is made up of individuals, so when we talk about the rules of team membership we are largely talking about what those individuals do. More specifically, it's about what they do in respect to themselves, and with respect to the wider team of stakeholders including the Scrum Master and Product Owner. So before we go any further, let's look at the behaviors that we can expect a disciplined agile developer to exhibit. What a good team member will do: Agree with other team members and the Product Owner to deliver a valuable and achievable piece of work every Sprint Understand the Sprint Backlog and how it correlates to the Sprint Goal Participate fully and actively in daily standups, planning sessions, reviews, and retrospectives Work with the rest of the team to meet each Sprint Goal (self-organize) Help other team members and the Product Owner to clarify requirements, such as by writing user stories and acceptance criteria Pro-actively remove skill silos, such as by pair programming or cross training, and without being told to do so Work with the Product Owner on an ongoing basis, so that work is understood, reviewed, and approved continually Make sure that the work done is transparent, such as by updating Scrum and Kanban boards Understand that they, and all team members, are stakeholders in the agile process Estimate work so that the Product Owner and other stakeholders can plan ahead (e.g. for release planning) Fully support and encourage the elicitation of metrics, and be able to interpret them and act on them Resolve outstanding or impeded work before actioning new work from the backlog Limit work in progress so as to maximize throughput Act immediately on impediments by appraising other team members and the Scrum Master of any issues, and help to resolve them Accept personal responsibility for the team's success Accept personal responsibility for their work meeting the team’s Definition of Done What a good team member doesn’t do… Fail to give the best unpadded estimates that can be provided at the time. Estimates should be given and received in good faith. Cherry pick work from the Sprint Backlog. The backlog is owned by the team and must be actioned in accordance with the team's Sprint Plan. Attempt to work on more than one item at a time. A good team member will pro-actively limit work in progress. Expect somebody else, such as a Scrum Master, to update the Scrum board or Kanban boards. Information radiators are owned by the team. Work in a "skills silo". A good team member does not view their work as a speciality that only he or she is able to work on. Claim that work has been completed if it does not satisfy the team’s Definition of Done Claim that work is complete if it does not meet the specific acceptance criteria that have been agreed for it Shot at Dawn: Teamwork and the Prime Directive If we were to apply the Patton philosophy in extremis, I suppose that an agile team would shoot its own malingerers following a retrospective, the Scrum Master standing by to deliver the coup-de-grace if needed. Although this lurid concept is absurd, how many experienced Scrum Masters have never secretly wished for a revolver in their desks, even for just a fleeting moment? It highlights a problem that the agile community is often evasive about. What should actually be done about a developer who causes problems for the rest of the team? Is it possible, or even desirable, to correlate the occurrences of those problems to the individual concerned? In a Sprint Retrospective, for example, no blame is ever meant to be directed towards any one team member. In fact the format of the session precludes the establishment of such a correlation, or even the inference that a particular individual may have been remiss in some way. Known as the "Prime Directive", this article of faith is meant to be recanted at the beginning of each retrospective session, and it has to be said in earnest. "Regardless of what we discover, we understand and truly believe that everyone did the best job he or she could, given what was known at the time, his or her skills and abilities, the resources available, and the situation at hand." The question is: what if we don't believe it though? What if all the evidence in the world is stacked against it? Should we go along with the directive anyway, and just kid ourselves for the duration of the session? If so, how can it possibly help? Where is the transparency, which we covet in agile practice, if we subscribe to this devil's credo that makes a mockery of the truth? The answer is potentially quite shocking, and certainly little understood. Don't think of the Prime Directive as a creed, or even as the temporary suspension of disbelief for the sake of the meeting. Think of it as a pre-condition that must hold, and genuinely be true, before a retrospective can happen at all. The underlying principle is that all of the attendees must be fully able to participate. All are expected to be professionals who can fulfil their duty to each other and to the Scrum process, and inspect and adapt their working practices accordingly. It isn't enough just to leave your knives at the door. You actually have to trust the people you are working with. Really trust them. Given that most developers are assigned to their teams by managers, and not by each other, this expectation of trust is indeed potentially shocking. It gets even scarier than that. Think about what all of this really means should trust be absent, or somehow lost. It means that you can't have a Sprint Retrospective at all until the issues around trust are resolved. It means that if a team member must be removed, then that should happen beforehand. Scrum does not go so far as to prescribe a mechanism for this, but it is established that a team will self-organize to remove its own problems. Perhaps they will have to make collective representations to a line manager, or petition for a member's removal through the Scrum Master. It might even mean that the team can deselect a team member by their own consensus. Yet however it is done, it appears that the team aren't too far removed from assembling a firing squad after all. If this all seems very draconian, let's reassert the key principle here: when Scrum is done properly a team will solve its own problems, including distasteful matters like this. Now, it has to be admitted that most Scrum teams across industry today don't get to operate at such a high level of proficiency. The consequences of this cut both ways. On the one hand a team may not be allowed to get on with their jobs without interference from management, while on the other hand they usually don't have to deal with the nastiness of putting a sick dog down. A few conversations with that same pointy-haired boss could be enough to get him to do the deed. Yet as the industry transitions more fully towards agile practice, this "remedy" will no longer be sustainable. Problems regarding a team member's competence won't be someone else's responsibility; rather, it will be incumbent upon the team to find a solution. In an agile world, greater responsibility falls on self-managing teams, along with their greater rights. Professionalism: from Team Discipline to Self Discipline In this article we've identified a range of behaviors that typify good team membership, and we've looked squarely at what should happen when things go wrong. In short, it's up to the team to sort out its own problems when a team member doesn't measure up. Yet this is only part of what disciplined agile practice is about. It isn't enough to put the focus on punitive measures and the threat of sanction, even if the exercising of authority is driven entirely by the team itself. What we need to do is to take things a step further. We don't really want discipline to be enforced by the team, even though they should be the ultimate arbiters. What we want is to encourage a self-discipline that wells up from each individual team member, and which serves as an inspiration to others. Disciplined teamwork isn't about empowerment. It's about cascading the release of potential through the clear demonstration of value. I look at it this way. There is only one person in this world any of us can change. I don't think I need to spell out who that person is. So, wherever you and your team may be on your agile journey, there should always be at least one person who can be relied upon. If that person does their bit, then they are helping to make the team more than the sum of its parts. "Don't empower me. Release me. I'll find my own power, and it will be far greater than anything you can bestow on me" - Tobias Mayer

[This blog was syndicated from http://blog.grallandco.com] Introduction When I was looking for a more effective way to create my cluster I asked some sysadmins which tools I should use to do it. The answer I got during OSDC was not Puppet, nor Chef, but wasAnsible. This article shows you how you can easily configure and create a Couchbase cluster deployed and many linux boxes...and the only thing you need on these boxes is an SSH Server! Thanks to Jan-Piet Mens that was one of the person that convinced me to use Ansible and answered questions I had about Ansible. You can watch the demonstration below, and/or look at all the details in the next paragraph. Ansible Ansible is an open-source software that allows administrator to configure and manage many computers over SSH. I won't go in all the details about the installation, just follow the steps documented in the Getting Started Guide. As you can see from this guide, you just need Python and few other libraries and clone Ansible project from Github. So I am expecting that you have Ansible working with your various servers on which you want to deploy Couchbase. Also for this first scripts I am using root on my server to do all the operations. So be sure you have register the root ssh keys to your administration server, from where you are running the Ansible scripts. Create a Couchbase Cluster So before going into the details of the Ansible script it is interesting to explain how you create a Couchbase Cluster. So here are the 5 steps to create and configure a cluster: Install Couchbase on each nodes of the cluster, as documented here. Take one of the node and "initialize" the cluster, using cluster-init command. Add the other nodes to the cluster, using server-add command. Rebalance, using rebalance command. Create a Bucket, using bucket-create command. So the goal now is to create an Ansible Playbook that executes these steps for you. Ansible Playbook for Couchbase The first think you need is to have the list of hosts you want to target, so I have create a hosts file that contains all my server organized in 2 groups: [couchbase-main] vm1.grallandco.com [couchbase-nodes] vm2.grallandco.com vm3.grallandco.com The group [couchbase-main] group is just one of the node that will drive the installation and configuration, as you probably already know, Couchbase does not have any master... All nodes in the cluster are identical. To ease the configuration of the cluster, I have create another file that contains all parameters that must be sent to all the various commands. This file is located in the group_vars/all see the section Splitting Out Host and Group Specific Data in the documentation. # Adminisrator user and password admin_user: Administrator admin_password: password # ram quota for the cluster cluster_ram_quota: 1024 # bucket and replicas bucket_name: ansible bucket_ram_quota: 512 num_replicas: 2 Use this file to configure your cluster. Let's describe the playbook file : - name: Couchbase Installation hosts: all user: root tasks: - name: download Couchbase package get_url: url=http://packages.couchbase.com/releases/2.0.1/couchbase-server-enterprise_x86_64_2.0.1.deb dest=~/. - name: Install dependencies apt: pkg=libssl0.9.8 state=present - name: Install Couchbase .deb file on all machines shell: dpkg -i ~/couchbase-server-enterprise_x86_64_2.0.1.deb As expected, the installation has to be done on all servers as root then we need to execute 3 tasks: Download the product, the get_url command will only download the file if not already present Install the dependencies with the apt command, the state=present allows the system to only install this package if not already present Install Couchbase with a simple shell command. (here I am not checking if Couchbase is already installed) So we have now installed Couchbase on all the nodes. Let's now configure the first node and add the others: - name: Initialize the cluster and add the nodes to the cluster hosts: couchbase-main user: root tasks: - name: Configure main node shell: /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-init-username=${admin_user} --cluster-init-password=${admin_password} --cluster-init-port=8091 --cluster-init-ramsize=${cluster_ram_quota} - name: Create shell script for configuring main node action: template src=couchbase-add-node.j2 dest=/tmp/addnodes.sh mode=750 - name: Launch config script action: shell /tmp/addnodes.sh - name: Rebalance the cluster shell: /opt/couchbase/bin/couchbase-cli rebalance -c 127.0.0.1:8091 -u ${admin_user} -p ${admin_password} - name: create bucket ${bucket_name} with ${num_replicas} replicas shell: /opt/couchbase/bin/couchbase-cli bucket-create -c 127.0.0.1:8091 --bucket=${bucket_name} --bucket-type=couchbase --bucket-port=11211 --bucket-ramsize=${bucket_ram_quota} --bucket-replica=${num_replicas} -u ${admin_user} -p ${admin_password} Now we need to execute specific taks on the "main" server: Initialization of the cluster using the Couchbase CLI, on line 06 and 07 Then the system needs to ask all other server to join the cluster. For this the system needs to get the various IP and for each IP address execute the add-server command with the IP address. As far as I know it is not possible to get the IP address from the main playbook YAML file, so I ask the system to generate a shell script to add each node and execute the script. This is done from the line 09 to 13. To generate the shell script, I use Ansible Template, the template is available in the couchbase-add-node.j2 file. {% for host in groups['couchbase-nodes'] %} /opt/couchbase/bin/couchbase-cli server-add -c 127.0.0.1:8091 -u ${admin_user} -p ${admin_password} --server-add={{ hostvars[host]['ansible_eth0']['ipv4']['address'] }:8091 --server-add-username=${admin_user} --server-add-password=${admin_password} {% endfor %} As you can see this script loop on each server in the [couchbase-nodes] group and use its IP address to add the node to the cluster. Finally the script rebalance the cluster (line 16) and add a new bucket (line 19). You are now ready to execute the playbook using the following command : ./bin/ansible-playbook -i ./couchbase/hosts ./couchbase/couchbase.yml -vv I am adding the -vv parameter to allow you to see more information about what's happening during the execution of the script. This will execute all the commands described in the playbook, and after few seconds you will have a new cluster ready to be used! You can for example open a browser and go to the Couchase Administration Console and check that your cluster is configured as expected. As you can see it is really easy and fast to create a new cluster using Ansible. I have also create a script to uninstall properly the cluster.. just launch ./bin/ansible-playbook -i ./couchbase/hosts ./couchbase/couchbase-uninstall.yml

You can easily tell Maven to include the version of the artifact and its Git/SVN/… revision in the JAR manifest file and then access that information at runtime via getClass().getPackage.getImplementationVersion(). (All credit goes to Markus Krüger and other colleagues.) Include Maven artifact version in the manifest (Note: You will actually not want to use it, if you also want to include a SCM revision; see below.) pom.xml: ... org.apache.maven.plugins maven-jar-plugin ... true true ... ... The resulting MANIFEST.MF of the JAR file will then include the following entries, with values from the indicated properties: Built-By: ${user.name} Build-Jdk: ${java.version} Specification-Title: ${project.name} Specification-Version: ${project.version} Specification-Vendor: ${project.organization.name Implementation-Title: ${project.name} Implementation-Version: ${project.version} Implementation-Vendor-Id: ${project.groupId} Implementation-Vendor: ${project.organization.name} (Specification-Vendor and Implementation-Vendor come from the POM’s organization/name.) Include SCM revision For this you can either use the Build Number Maven plugin that produces the property ${buildNumber}, or retrieve it from environment variables passed by Jenkinsor Hudson (SVN_REVISION for Subversion, GIT_COMMIT for Git). For git alone, you could also use the maven-git-commit-id-plugin that can either replace strings such as ${git.commit.id} in existing resource files (using maven’s resource filtering, which you must enable) with the actual values or output all of them into a git.properties file. Let’s use the buildnumber-maven-plugin and create the manifest entries explicitely, containing the build number (i.e. revision) org.codehaus.mojo buildnumber-maven-plugin 1.2 validate create false false org.apache.maven.plugins maven-jar-plugin 2.4 ${project.name} ${project.version} ${buildNumber} ... Accessing the version & revision As mentioned above, you can access the manifest entries from your code via getClass().getPackage.getImplementationVersion() andgetClass().getPackage.getImplementationTitle(). References SO: How to get Maven Artifact version at runtime? Maven Archiver documentation

In this blog post, I will present a simple tutorial on uploading a large file to Amazon S3 as fast as the network supports. Amazon S3 is clustered storage service of Amazon. It is designed to make web-scale computing easier. Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers. For using Amazon services, you'll need your AWS access key identifiers, which AWS assigned you when you created your AWS account. The following are the AWS access key identifiers: Access Key ID (a 20-character, alphanumeric sequence) For example: 022QF06E7MXBSH9DHM02 Secret Access Key (a 40-character sequence) For example: kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct Caution Your Secret Access Key is a secret, which only you and AWS should know. It is important to keep it confidential to protect your account. Store it securely in a safe place. Never include it in your requests to AWS, and never e-mail it to anyone. Do not share it outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your Secret Access Key. The Access Key ID is associated with your AWS account. You include it in AWS service requests to identify yourself as the sender of the request. The Access Key ID is not a secret, and anyone could use your Access Key ID in requests to AWS. To provide proof that you truly are the sender of the request, you also include a digital signature calculated using your Secret Access Key. The sample code handles this for you. Your Access Key ID and Secret Access Key are displayed to you when you create your AWS account. They are not e-mailed to you. If you need to see them again, you can view them at any time from your AWS account. To get your AWS access key identifiers Go to the Amazon Web Services web site at http://aws.amazon.com. Point to Your Account and click Security Credentials. Log in to your AWS account. The Security Credentials page is displayed. Your Access Key ID is displayed in the Access Identifiers section of the page. To display your Secret Access Key, click Show in the Secret Access Key column. You can use your Amazon keys from a properties file in your application. Here is a sample for properties file containing Amazon keys: # Fill in your AWS Access Key ID and Secret Access Key # http://aws.amazon.com/security-credentials accessKey = secretKey = Here is sample AmazonUtil class for getting AWS Credentials from properties file. public class AmazonUtil { private static final Logger logger = LogUtil.getLogger(); private static final String AWS_CREDENTIALS_CONFIG_FILE_PATH = ConfigUtil.CONFIG_DIRECTORY_PATH + File.separator + "aws-credentials.properties"; private static AWSCredentials awsCredentials; static { init(); } private AmazonUtil() { } private static void init() { try { awsCredentials = new PropertiesCredentials(IOUtil.getResourceAsStream(AWS_CREDENTIALS_CONFIG_FILE_PATH)); } catch (IOException e) { logger.error("Unable to initialize AWS Credentials from " + AWS_CREDENTIALS_CONFIG_FILE_PATH); } } public static AWSCredentials getAwsCredentials() { return awsCredentials; } } Amazon S3 has Multipart Upload service which allows faster, more flexible uploads into Amazon S3. Multipart Upload allows you to upload a single object as a set of parts. After all parts of your object are uploaded, Amazon S3 then presents the data as a single object. With this feature you can create parallel uploads, pause and resume an object upload, and begin uploads before you know the total object size. For more information on Multipart Upload, review the Amazon S3 Developer Guide In this tutorial, my sample application uploads each file parts to Amazon S3 with different threads for using network throughput as possible as much. Each file part is associated with a thread and each thread uploads its associated part with Amazon S3 API. Figure 1. Amazon S3 Parallel Multi-Part File Upload Mechanism Amazon S3 API suppots MultiPart File Upload in this way: 1. Send a MultipartUploadRequest to Amazon. 2. Get a response containing a unique id for this upload operation. 3. For i in ${partCount} 3.1. Calculate size and offset of split-i in whole file. 3.2. Build a UploadPartRequest with file offset, size of current split and unique upload id. 3.3. Give this request to a thread and starts upload by running thread. 3.3.1. Send associated UploadPartRequest to Amazon. 3.3.2. Get response after successful upload and save ETag property of response. 4. Wait all threads to terminate 5. Get ETags (ETag is an identifier for successfully completed uploads) of all terminated threads. 6. Send a CompleteMultipartUploadRequest to Amazon with unique upload id and all ETags. So Amazon joins all file parts as target objects. Here is implementation: public class AmazonS3Util { private static final Logger logger = LogUtil.getLogger(); public static final long DEFAULT_FILE_PART_SIZE = 5 * 1024 * 1024; // 5MB public static long FILE_PART_SIZE = DEFAULT_FILE_PART_SIZE; private static AmazonS3 s3Client; private static TransferManager transferManager; static { init(); } private AmazonS3Util() { } private static void init() { // ... s3Client = new AmazonS3Client(AmazonUtil.getAwsCredentials()); transferManager = new TransferManager(AmazonUtil.getAwsCredentials()); } // ... public static void putObjectAsMultiPart(String bucketName, File file) { putObjectAsMultiPart(bucketName, file, FILE_PART_SIZE); } public static void putObjectAsMultiPart(String bucketName, File file, long partSize) { List partETags = new ArrayList(); List uploaders = new ArrayList(); // Step 1: Initialize. InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, file.getName()); InitiateMultipartUploadResult initResponse = s3Client.initiateMultipartUpload(initRequest); long contentLength = file.length(); try { // Step 2: Upload parts. long filePosition = 0; for (int i = 1; filePosition < contentLength; i++) { // Last part can be less than part size. Adjust part size. partSize = Math.min(partSize, (contentLength - filePosition)); // Create request to upload a part. UploadPartRequest uploadRequest = new UploadPartRequest(). withBucketName(bucketName).withKey(file.getName()). withUploadId(initResponse.getUploadId()).withPartNumber(i). withFileOffset(filePosition). withFile(file). withPartSize(partSize); uploadRequest.setProgressListener(new UploadProgressListener(file, i, partSize)); // Upload part and add response to our list. MultiPartFileUploader uploader = new MultiPartFileUploader(uploadRequest); uploaders.add(uploader); uploader.upload(); filePosition += partSize; } for (MultiPartFileUploader uploader : uploaders) { uploader.join(); partETags.add(uploader.getPartETag()); } // Step 3: complete. CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName, file.getName(), initResponse.getUploadId(), partETags); s3Client.completeMultipartUpload(compRequest); } catch (Throwable t) { logger.error("Unable to put object as multipart to Amazon S3 for file " + file.getName(), t); s3Client.abortMultipartUpload( new AbortMultipartUploadRequest( bucketName, file.getName(), initResponse.getUploadId())); } } // ... private static class UploadProgressListener implements ProgressListener { File file; int partNo; long partLength; UploadProgressListener(File file) { this.file = file; } @SuppressWarnings("unused") UploadProgressListener(File file, int partNo) { this(file, partNo, 0); } UploadProgressListener(File file, int partNo, long partLength) { this.file = file; this.partNo = partNo; this.partLength = partLength; } @Override public void progressChanged(ProgressEvent progressEvent) { switch (progressEvent.getEventCode()) { case ProgressEvent.STARTED_EVENT_CODE: logger.info("Upload started for file " + "\"" + file.getName() + "\""); break; case ProgressEvent.COMPLETED_EVENT_CODE: logger.info("Upload completed for file " + "\"" + file.getName() + "\"" + ", " + file.length() + " bytes data has been transferred"); break; case ProgressEvent.FAILED_EVENT_CODE: logger.info("Upload failed for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; case ProgressEvent.CANCELED_EVENT_CODE: logger.info("Upload cancelled for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; case ProgressEvent.PART_STARTED_EVENT_CODE: logger.info("Upload started at " + partNo + ". part for file " + "\"" + file.getName() + "\""); break; case ProgressEvent.PART_COMPLETED_EVENT_CODE: logger.info("Upload completed at " + partNo + ". part for file " + "\"" + file.getName() + "\"" + ", " + (partLength > 0 ? partLength : progressEvent.getBytesTransfered()) + " bytes data has been transferred"); break; case ProgressEvent.PART_FAILED_EVENT_CODE: logger.info("Upload failed at " + partNo + ". part for file " + "\"" + file.getName() + "\"" + ", " + progressEvent.getBytesTransfered() + " bytes data has been transferred"); break; } } } private static class MultiPartFileUploader extends Thread { private UploadPartRequest uploadRequest; private PartETag partETag; MultiPartFileUploader(UploadPartRequest uploadRequest) { this.s3Client = s3Client; this.uploadRequest = uploadRequest; } @Override public void run() { partETag = s3Client.uploadPart(uploadRequest).getPartETag(); } private PartETag getPartETag() { return partETag; } private void upload() { start(); } } }

There are many good ideas and practices in Agile development, ideas and practices that definitely work: breaking projects into Small Releases to manage risk and accelerate feedback; time-boxing to limit WIP and keep everyone focused; relying only on working software as the measure of progress; simple estimating and using velocity to forecast team performance; working closely and constantly with the customer; and Continuous Integration – and Continuous Delivery – to ensure that code is always working and stable. But there are other commonly accepted ideas and best practices that aren’t important: if you don’t follow them, nothing bad will happen to you and your project will still succeed. And there are a couple that you are better off not following at all. Test-Driven Development Teams that need to move quickly need to depend on a fast, efficient testing safety net. With Test First Development or Test-Driven Development (TDD), there’s no excuse for not writing tests – after all, you have to write a failing test before you write the code. So you end up with a good set of working automated tests that ensure a high level of coverage and regression protection. TDD is not only a way of ensuring that developers test their code. It is also advocated as a design technique that leads to better quality code and a simpler, cleaner design. A study of teams at Microsoft and IBM (Realizing Quality Improvement through Test Driven Development, Microsoft Research, 2008) found that while TDD increased upfront development costs between 15-35% (TDD demands developers change the way that they think and work, which slows developers down, at least at first), it reduced defect density by 40% (IBM) or as much as 60-90% (Microsoft) over teams that did not follow disciplined unit testing. But in Making Software Chapter 12 “How Effective is Test-Driven Development” researchers led by Burak Turhan found that while TDD improves external quality (measured by one or more of test cases passed, number of defects, defect density, defects per test, effort required to fix defects, change density, % of preventative changes) and can improve the quality of the tests (fewer mistakes in the tests, tests that are easier to maintain), TDD does not consistently improve the quality of the design. TDD seems to reduce code complexity and improve reuse, however it also negatively impacts coupling and cohesion. And while method and class-level complexity is better in code developed using TDD, project/package level complexity is worse. People who like TDD like it a lot, so if you like it, do it. And even if you are not TDD-infected, there are times when working test first is natural – when you have to solve a specific problem in a specific way, or if you’re fixing a bug where the failing test case is already written up for you. But the important thing is that you write a good set of tests and keep them up to date and run them frequently – it doesn't matter if you write them before, or after, you write the code. Pair Programming According to the VersionOne State of Agile Development Survey 2012, almost 1/3 of teams follow pair programming – a surprisingly high number, given how disciplined pair programming is, and how few teams follow XP (2%) or Scrum/XP Hybrid (11%) methods where pair programming would be prescribed. There are good reasons for pairing: information sharing and improving code quality through continuous, informal code reviews as developers work together. And there are natural times to pair developers, or sometimes developers and testers, together: when you’re working through a hard design problem; or on code that you’ve never seen before and somebody who has worked on it is available to help; or when you’re over your head in troubleshooting a high-pressure problem; or testing a difficult part of the system; or when a new person joins the team and needs to learn about the code and coding practices. Some (extroverted) people enjoy pairing up, the energy it creates and the opportunities it provides to get to know others on the team. But forcing people who prefer working on their own or who don’t like each other to work closely together is definitely not a good idea. There are real social costs in pairing: you have to be careful to pair people up by skill, experience, style, personality type and work ethic. And sustained pair programming can be exhausting, especially over the long term – one study (Vanhanen and Lassenius 2007) found that people only pair between 1.5 and 4 hours a day on average, because it’s too intense to do all day long. In Pair Programming Considered Harmful? Jon Evans says that pairing can have also negative effects on creativity: Research strongly suggests that people are more creative when they enjoy privacy and freedom from interruption … What distinguished programmers at the top-performing companies wasn’t greater experience or better pay. It was how much privacy, personal workspace and freedom from interruption they enjoyed,” says a New York Times article castigating “the new groupthink”. And in “Still Questioning Extreme Programming” Pete McBreen points out some other disadvantages and weaknesses of pair programming: Exploration of ideas is not encouraged, pairing makes a developer focus on writing the code, so unless there is time in the day for solo exploration the team gets a very superficial level of understanding of the code. Developers can come to rely too much on the unit tests, assuming that if the tests pass then the code is OK. (This follows on from the lack of exploration.) Corner cases and edge cases are not investigated in detail, especially if they are hard to write tests for. Code that requires detail thinking about the design is hard to do when pairing unless one partner completely dominates the session. With the usual tradeoff between partners, it is hard to build technically complex designs unless they have been already been worked out in a solo session. Personal styles matter when pairing, and not all pairings are as productive as others. Pairs with different typing skills and proficiencies often result in the better typist doing all of the coding with the other partner being purely passive. And of course pairing in distributed teams doesn't work well if at all (depending on distance, differences in time zones, culture, working styles, language), although some people still try. While pairing does improve code quality over solo programming, you can get the same improvements in code quality, and at least some of the information sharing advantages, through code reviews, at less cost. Code reviews – especially lightweight, offline reviews – are easier to schedule, less expensive and less intrusive than pairing. And as Jason Cohen points out even if developers are pair programming, you may still need to do code reviews, because pair programming is really about joint problem solving, and doesn’t cover all of the issues that a code review would. Back to Jon Evans for the final word on pair programming: The true answer is that there is no one answer; that what works best is a dynamic combination of solitary, pair, and group work, depending on the context, using your best judgement. Paired programming definitely has its place. (Betteridge’s Law strikes again!) In some cases that place may even be “much of most days.” But insisting on 100 percent pairing is mindless dogma, and like all mindless dogma, ultimately counterproductive. Emergent Design and Metaphor Incremental development works, and trying to keep design simple makes good sense, but attempting to define an architecture on the fly is foolish and impractical. There’s a reason that almost nobody actually follows Emergent Design: it doesn't work. Relying on a high-level metaphor (the system is an "assembly line" or a "bill of materials" or a "hive of bees") shared by the team as some kind of substitute for architecture is even more ridiculous. Research from Carnegie Mellon University found that … natural language metaphors are relatively useless for either fostering communication among technical and non-technical project members or in developing architecture. Almost no one understands what a system metaphor is any ways, or how it is to be used, or how to choose a meaningful metaphor or how to change it if you got it wrong (and how you would know if you got it wrong), including one of the people who helped come up with the idea: Okay I might as well say it publicly - I still haven't got the hang of this metaphor thing. I saw it work, and work well on the C3 project, but it doesn't mean I have any idea how to do it, let alone how to explain how to do it. Martin Fowler, Is Design Dead? Agile development methods have improved development success and shown better ways to approach many different software development problems – but not architecture and design. Daily Standups When you have a new team and everyone needs to get to know each other and more time to understand what the project is about; or when the team is working under emergency conditions trying to fix something or finish something under extreme pressure, then getting everyone together in regular meetings, maybe even more than once a day, is necessary and valuable. But whether everyone stands up or sits down and what they end up talking about in a meeting should be up to you. If your team has been working well together for a while and everyone knows each other and knows what they are working on, and if developers update cards on a task board or a Kanban board or the status in an electronic system as they get things done, and if they are grown up enough to ask for help when they need it, then you don’t need to make them all stand up in a room every morning. Collective Code Ownership Letting everyone work on all of the code isn't always practical (because not everyone on the team has the requisite knowledge or experience to work on every problem) and collective code ownership can have negative effects on code quality. Share code where it makes sense to do so, but realize that not everybody can – or should – work on every part of the system. Writing All Requirements as Stories The idea that every requirement specification can be written as User Stories in 1 or 2 lines on cards, that requirements should be too short on purpose (so that the developer has to talk to someone to explain what’s really needed) and insisting that they should all be in the same template form “As a type of user I want some goal so that some reason…” is silly and unnecessary. This is the same kind of simple minded orthodoxy that led everyone to try to capture all requirements in UML Use Case format with stick men and bubbles 15 years ago. There are many different ways to effectively express requirements. Sometimes requirements need to be specified in detail (when you have to meet regulatory compliance or comply with a standard or integrate with an existing system or implement a specific algorithm or…). Sometimes it’s better to work from a test case or a detailed use case scenario or a wire frame or some other kind of model, because somebody who knows what’s going on has already worked out the details for you. So pick the format and level of detail that works best and get to work. Relying on a Product Owner Relying on one person as the Product Owner, as the single solitary voice of the customer and the “one throat to choke” when the project fails, doesn't scale, doesn't last, and puts the team and the project and eventually the business at risk. It’s a naïve, dangerous approach to designing a product and to managing a development project, and it causes more problems than it solves. Many teams have realized this and are trying to work around the Product Owner idea because they have to. To succeed, a team needs real and sustained customer engagement at multiple levels, and they should take responsibility themselves for making sure that they get what they need, rather than relying on one person to do it all.

Github has a nice API for inspecting repositories – it lets you read gists, issues, commit history, files and so on. Git repository data lends itself to demonstrating the power of combining full text and faceted search, as there is a mix of free text fields (commit messages, code) and enumerable fields (committers, dates, committer employers). Github APIs return JSON, which has the nice property of resembling a tree structure – results can be recursed over without fear of infinite loops. Note that to download the entire commit history for a repository, you need to page through it by sha hash. The API I use here lacks diffs, which must be retrieved elsewhere. To test this, access a URL like so. The configurable arguments are the repository owner and name fields. https://api.github.com/repos/torvalds/linux/commits This is what a commit looks like: { "sha": "7638417db6d59f3c431d3e1f261cc637155684cd", "url": "https://api.github.com/repos/octocat/Hello-World/git/commits/7638417db6d59f3c431d3e1f261cc637155684cd", "author": { "date": "2008-07-09T16:13:30+12:00", "name": "Scott Chacon", "email": "[email protected]" }, "committer": { "date": "2008-07-09T16:13:30+12:00", "name": "Scott Chacon", "email": "[email protected]" }, "message": "my commit message", "tree": { "url": "https://api.github.com/repos/octocat/Hello-World/git/trees/827efc6d56897b048c772eb4087f854f46256132", "sha": "827efc6d56897b048c772eb4087f854f46256132" }, "parents": [ { "url": "https://api.github.com/repos/octocat/Hello-World/git/commits/7d1b31e74ee336d15cbd21741bc88a537ed063a0", "sha": "7d1b31e74ee336d15cbd21741bc88a537ed063a0" } ] } To make the test simple, I download these as JSON locally, then start a python webserver. Were I to make many such calls on a public site, I’d set up a proxy to the github APIs. python -m SimpleHTTPServer This data has a number of nested objects and must be flattened to fit into the lunr.jsfull-text index. This example uses the commit number (0, 1, 2..N) as the location in the index, but a real environment should use the commit hash to allow partitioning the ingestion process. Nested objects are flattened by joining subsequent keys with underscores in between. A production-worthy solution needs to escape these to prevent collisions. var documents = []; function recurse(doc_num, base, obj, value) { if ($.isPlainObject(value)) { $.each(value, function (k, v) { recurse(doc_num, base + obj + "_", k, v); }); } else { process(doc_num, base + obj, value); } } function process(doc_num, key, value) { if (documents.length <= doc_num) documents[doc_num] = {}; if (value !== null) documents[doc_num][key] = value + ''; } $.each(data, function(doc_num, commit) { $.each(commit, function(k, v) { recurse(doc_num, '', k, v) }); }); Normally, one sets up a lunr full-text index by specifying all the fields, much like Solr’s numerous XML config files. Lunr doesn’t have nearly as many configuration options, since you only specify the ‘boost’ parameter to increase the value of certain fields in ranking. I imagine this will change as the project grows, at the very least to include type hints. Given the simplicity of field objects, you can infer infer the field list from JSON payloads. The code below provides two modes, one where you inspect the entire JSON payload, or one where you limit how many commits you check, a good option when JSON data is consistent. The function accepts configuration objects resembling ExtJS config objects, which lets you override as desired. If fields derived from existing data are required, they can be inserted after any documents are inserted. function inferIndex(documents, config) { return lunr(function() { this.ref('id'); var found = {}; var idx = this; $.each(documents, function(doc_num, doc) { if (config && config.limit && config.limit < doc_num) return; $.each(doc, function(k, v) { if (!found[k]) { if (config && config[k]) { idx.field(k, config[k]); } else { idx.field(k); } found[k] = true; } }); }); }); } var index = inferIndex(documents, {limit: 1, 'commit_author_name':{boost:10}); Inserting flattened documents into the index becomes simple. The method below provides a callback, should you desire to add calculated fields fields. $.each(documents, function(doc_num, attrs, doc_cb) { var doc = $.extend( {id: doc_num}, attrs); if (doc_cb) { doc = doc_cb(doc); } index.add(doc); }); At this point we’ve indexed the entire commit history from a git repository, which lets us search for commits by topic. While this is useful, it’d be really nice to be able to facet on fields, which would return the number of documents in a category, like a SQL group by. I’ve found it particularly convenient to facet on author, date, or author’s company. If you have access to the original documents, you can easily construct facets based on the results of a lunr search: function facet(index, query, data, field) { var results = index.search(query); var facets = {}; $.each(results, function(index, searchResult) { var doc = data[searchResult.ref]; facets[doc[field]] = (facets[doc[field]] === undefined ? 0 : facets[doc[field]]) + 1; } ); return facets; } Commit messages in repositories where I work often contain names of clients who requested a feature or bug fix. Consequently doing a search faceted by author provides a list of who worked with each client the most – this can also tell you who has worked with various pieces of technology. The following query demonstrates this technique: var facets = facet(index, 'driver', documents, 'commit_author_name'); {"Wolfram Sang":24,"Linus Torvalds":3} The approach shown here works well, but requires retrieving results requires access to the original document data. If we want to filter the results to a category, we need a richer search API than lunr currently provides, as well as callback options within the search API. In Solr there are also options to skip lower-casing data, as that may be inappropriate for category titles. Mitigating these issues will be explored further in future essays.

If you’re uploading blobs by splitting blobs into blocks and you get the error – The specified blob or block content is invalid, then this post is for you. Short Version If you’re uploading blobs by splitting blobs into blocks and you get the above mentioned error, ensure that your block ids of your blocks are of same length. If the block ids of your blocks are of different length, you’ll get this error. Long Version Now for the longer version of this post . A few days back I was working with storage client library especially around uploading blobs in chunks and with one particular blob I was constantly getting the error – The specified blob or block content is invalid. I tried numerous combinations even resorting to REST API directly but to no avail. It only happened with just one blob. Furthermore if I uploaded the same blob without splitting it into blocks, all was well. I was at my wits’ end. Tried searching the Internet for this error but could not find a conclusive answer to my problem. After much trial and error, I was able to simulate the same problem on other blobs as well. Here’s how you can recreate it: Start uploading the blob by splitting it into blocks. For block id, let’s do a 7 character long string e.g. intValue.ToString(“d7”). This will ensure that my block ids would be “0000001”, “0000002”, …, ”0000010” ….. After one or two blocks are uploaded, cancel the operation. Now re-upload the blob by splitting it into blocks. However this time for block id, let’s do a 6 character long string e.g. intValue.ToString(“d6”). You’ll get the error as soon as you try to upload the 1st block. Possible Solutions Now that we know the root cause of this problem, let’s look at some of the possible solutions to solve this problem. Wait out One possible solution is to wait out. I know its lame but still a possible solution. We know that Windows Azure Blob Storage Service keeps all uncommitted blocks for a duration of 7 days and if within 7 days those uncommitted blocks are not committed, the storage service purges them. I wish storage service provided some mechanism to purge uncommitted blocks programmatically. Commit uncommitted blocks You could possibly commit the blocks which are in uncommitted state so that at least you get a blob (which would not be the blob we wanted to upload in the first place). You can then delete that blob and re-upload the blob by specifying block ids which are of same length. To fetch the list of uncommitted blocks, if you’re using REST API directly you can perform “Get Block List” operation and pass “blocklisttype=uncommitted” as one of the query string parameters. If you’re using storage client library (assuming you’re using the version 2.x of .Net storage client library), you can do something like the code below: private static List GetUncommittedBlockIds(CloudBlockBlob blob) { var sasUri = blob.GetSharedAccessSignature(new SharedAccessBlobPolicy() { SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(5), Permissions = SharedAccessBlobPermissions.Read, }); var blobUri = new Uri(string.Format("{0}{1}", blob.Uri, sasUri)); List uncommittedBlockIds = new List(); var request = BlobHttpWebRequestFactory.GetBlockList(blobUri, null, null, BlockListingFilter.Uncommitted, null, null); //request.Headers.Add("Authorization", using (var resp = (HttpWebResponse)request.GetResponse()) { using (var stream = resp.GetResponseStream()) { var getBlockListResponse = new GetBlockListResponse(stream); var blocks = getBlockListResponse.Blocks; foreach (var block in blocks.Where(b => !b.Committed)) { uncommittedBlockIds.Add(Encoding.UTF8.GetString(Convert.FromBase64String(block.Name))); } } } return uncommittedBlockIds; } A few things to keep in mind here: Microsoft.WindowsAzure.Storage.Blob namespace does not have the capability to get the list of uncommitted blocks. You would need to make use ofMicrosoft.WindowsAzure.Storage.Blob.Protocol namespace. Because we’re kind of invoking the REST API by executing an HttpWebRequest, I created a shared access signature on the blob so that I don’t have to create “Authorization” header. Fetch uncommitted blocks to see block id length You could fetch the list of uncommitted blocks just to find out the length of the block id used. You could then use that block id length for your new upload session and do the upload. Please see the code snippet above to find this information. Upload another blob with same name without splitting it into blocks You could also upload another blob with the same name without splitting it into blocks. It could very well be a zero byte blob. That way your uncommitted block list will be wiped clean. Then you could delete that dummy blob and re-upload the actual blob. A Few Words About Blocks Since we’re talking about blocks, I thought it might be useful to mention a few points about them: Blocks and block related operations are only applicable for “Block Blobs”. Duh!! You’ll get an error if you’re trying to do these operations on a “Page Blob”. For uploading large blobs, it is recommended that you split your blob into blocks. In fact if your blob size is more than 64 MB, then you have to split it into blocks. Minimum size of a block is 1 Byte and the maximum size of a block is 4 MB. It is recommended that you choose a block size based on your internet connectivity and number of parallel threads you want use to upload these blocks. A blob can be split into a maximum of 50000 blocks. It’s important to remember this limitation because you are reminded of this limit when you’re trying to upload 50001st block. The length of all the block ids must be same. So if you’re using an integer value to denote block id, you make sure that you pad that integer value with “0” so that you get same length. So you could do something likeint.ToString(“d6”). When passing the block id as a parameter, it must be Base64 encoded. While the order in which the blocks are uploaded is not important, the order is important when you commit the block list because that’s when the blob is constructed by the service. For example, let’s say you’re uploading a blob by splitting it into 5 blocks (with ids “000001”, “000002”, “000003”, “000004”, and “000005”). You could upload these blocks in any order – 000004, 000001, 000003, 000005, 000002 however when you commit the block list, ensure that the block ids are passed in proper order i.e. 000001, 000002, 000003, 000004, 000005. Summary That’s it for this post. I hope you’ve found this information useful. I spent considerable amount of time trying to fix this problem so I hope it will help some folks out. As always, if you find any issues with the post please let me know and I’ll fix it ASAP.

Learn how to run test code with Gradle using the test task that is added by the Java plugin.

this month, my fellow it pro technical evangelists and i are authoring a new series of articles on 20 key scenarios with windows azure infrastructure services . check out the list of articles here: http://mythoughtsonit.com/2013/05/20-key-scenarios-with-windows-azure-infrastructure-services/ . web-based distributed authoring and versioning, or webdav, is a set of protocols based on http that allows end-users to map a network drive over http and edit content and files stored on the web server. when webdav was first offered on microsoft server i had evaluated it and decided it did not perform well enough for me. the webdav extension to iis was completely rewritten back in the server 2008 timeframe and is worth taking a look at again. in this article i will guide you step by step through the process of setting up webdav on server 2012 in a windows azure iaas environment. this will give you a solid performing file share on the internet over port 80 and the http protocol. first you need an azure account. you can setup a free trail of azure. details can be found here: http://mythoughtsonit.com/2013/04/step-by-step-guide-to-setting-up-a-windows-azure-free-trial/ second provision a server 2012 machine. watch a video of what to do here: third open port 80 to this new server: in the azure portal select your 2012 server and choose the “endpoints” tab on the top. click “add endpoint” at the bottom of the screen enter the endpoint information for port 80 to port 80 done. next we need to install the iis webserver and webdav. installing webdav on iis 8.0 start server manager and go to “add roles and features” under server roles – add the web server (iis) role click through the wizard until you come to the role services section. then find and select “webdav publishing” and “windows authentication” click next and then install when the install is finished you are ready to move on to the next section. configuring iis 8 for webdav after the installation finishes you need to configure the box for access. start the iis manager tool. choose the “default web site” on the left side. then click on “authentication” open the windows authentication option and enable it. open the “webdav authoring rules” create a webdav rule. i choose to allow all users access to all content. a better security practice is to limit what users can use the service. it’s your data so you decide. make sure webdav is enabled and that your access rule is set: that is it… now your ready to access your webdav file share! test and insure you can hit the web server by using your browser: because you opened port 80 and installed iis 8 you should see the default web page when you browse to your servers internet dns name. example: http://yourdomainname.cloudapp.net/ how to map a drive to your webdav server: there are two ways i use to connect to the webdav server how to map a drive to your webdav server from the win 8 gui: from windows explorer, right click on “computer” and select “map a network drive” map your network drive by entering the address to your server example: http://yourdomainname.cloudapp.net/ i selected “connect using different credentials” because my workstation was not joined to the server in anyway and i needed to use an account in the servers local sam database. hit “finish” and enter your credentials. now you will have a connected drive that you can access from windows explorer or any tool via the drive mapping. how to map a drive to your webdav server from a cmd box: 1. hit windows start and type: cmd 2. enter the command: net use [drive letter] [url] example: net use e: http://yourdomainname.cloudapp.net/

A couple of weeks ago we looked at how to do a quick "health check" of an agile team. We saw that a great deal can be learned just by attending one of their daily stand-ups and by inspecting the state of their Scrum and Kanban boards. Of course these are nothing more than cursory examinations, and serious ailments can lie behind an apparently robust façade of agile practice. If you have reason to believe that a team is dysfunctional, you might have to dig deeper than the superficial evidence suggested by its apparent morphology. In my experience the next thing to examine is the team's "Definition of Done". This is the standard to which all work is put before it can be considered to be complete. Each team is collectively responsible for its own Definition of Done. It's up to them to make sure that it is adequate, and that it is applied by all members to all of the work they do. Without such professional oversight there can be no assurance that any deliverable will truly be fit for release. A spiffy stand-up and a cracker of a Kanban board might suggest rude team health, but they are no guarantee that the Definition of Done is solid, or that it isn't being undercut by someone along the way. Technical debt and rework are the main symptoms to look for. The consequences of backsliding on a Definition of Done might not become apparent until long after the events that caused it. By then, that rework or debt can be difficult to trace to the specific behaviors of those who cheated the system. You see, unfortunately a Definition of Done is a bit like personal hygiene. If there is no oversight, everyone can pretend that they are following the rules for the sake of the team, even though the presence of E. Coli on the office keyboards will tell its own tale about compliance. Everyone knows that it has to be coming from somewhere, but won't admit to their own liability or involvement, perhaps not even to themselves. Just as team vomiting will follow one member's dubious hand-washing practices, a short-changed Definition of Done will lead to rework by the team or the creation of technical debt. This is why team ownership and enforcement of what "Done" means is so important. An effective Definition of Done has to be founded on a healthy balance between due diligence and professional trust. What does this mean for agile development? You can think of a Definition of Done as the key defensive bulwark in software development epidemiology. If you balance the right level of team oversight with the right level of trust, severe outbreaks of technical debt or rework will become rare. High levels of oversight may be needed to start with, since team members might not have bought in to the idea of "done" yet. Once people become conditioned to do the right thing and see themselves as stakeholders in the team and its success, the balance can swing more towards trust. People become reluctant to renege on a team investment if they can see that it adds value for everyone including them. What's more, a Definition of Done improves the more it is respected, and becomes better respected the more it improves. In terms of agile best practice a Definition of Done will be used to determine whether or not User Story implementations are release-ready. However, each team can implement many User Stories over the course of a sprint, and making sure that all of these stories meet the Definition of Done can be challenging. Teams that are new to agile methods often have more modest ambitions. For example, their Definition of Done may only extend as far as delivery into a pre-production environment. Of course, anything less than "fully release ready" incurs the risk of technical debt and the need to pay it back later. Yet like a sloppy approach to hand-washing, it has to be admitted that something is better than nothing at all. Applying a Definition of Done consistently to even a sub-optimal standard will at least permit the delivery of each User Story to a known level of quality. It might not be great but at least it's there, and it's something that can be built upon and improved. The Lessons of Lean-Kanban Lean-Kanban methodologies have an instructive relationship with the Definition of Done. In these approaches the optimization of the value stream is of great significance. Naturally though, if a value stream is to be optimized it must first be understood. This means breaking the stream down into multiple discrete steps that can be studied for bottlenecks and any other occurrences of waste. For example, "Work in Progress" can be broken down into finer-grained stations such as "In Development", "Peer Review", "QA Test", "Knowledge Transfer", and "In Deployment". Team members will be cross-trained and will move freely across those stations in order to expedite as smooth a workflow as possible. Now here's where it gets interesting. If a Lean-Kanban operation has multiple well-defined stations, the case for having a Definition of Done can seem rather harder to make. After all, by the time a User Story gets to "Done", you already know that it has gone through the key steps you care about in the development process. What value can a Definition of Done really add in such a situation? Doesn't it just become waste itself? To find the answer, we need to look back to the manufacturing roots of Lean-Kanban. In a car plant for example, the steps of construction are exceptionally well defined and team members can move freely over several dozen stations. Some of those stations will be for the chassis, others for the interior, others for the engine block and electrics and so on. Yet despite this the Definition of Done will be an absolute corker, and much of the process of verification will be automated. Each station might even have its own Definition of Done so inspection can occur as close as possible to the point of action and potential remedy. The total number of checks that happen before each vehicle leaves the factory will be exhaustive. Why is this thought to be necessary? Because the manufacturers know perfectly well that the verification of "done" adds value. Merely having well-defined stations is no guarantee that everything will be done well. Quality is built in and validated by inspection. One thing's for sure: no-one in IT should accuse car manufacturers of having a weak understanding of what "done" means. The Definition of Done versus Acceptance Criteria However, software projects have a wild-card to deal with that car manufacturers don't have to worry about. Unlike the car doors and carburettors that roll down an assembly line, each User Story is different and has to be treated as a special case. To deal with this, each User Story has Acceptance Criteria that are agreed by the team members and the Product Owner as part of a Sprint Planning Session. Acceptance Criteria have to be quite specific to particular User Stories, because each story can be unique. The Definition of Done, on the other hand, applies to all of the User Stories being worked on by a team. The associated conditions must be invariant. For example, if all work has to be peer reviewed and subjected to QA testing prior to release, then those criteria would be enumerated in the Definition of Done rather than being repeated in each User Story's Acceptance Criteria. If the definition is enforced properly, a developer could never claim that a User Story was “Done” if it hadn't both been reviewed and passed QA testing. Writing a Definition of Done The Scrum Guide describes a Definition of Done as a "shared understanding of what it means for work to be complete". No process is suggested for writing a Definition of Done, nor in fact is there any suggestion that one should be written down at all. However, a documented definition may go some way towards providing that shared understanding. Here's how you can set about eliciting one: Review Acceptance Criteria: Gather the Acceptance Criteria for work completed so far Look for common criteria that can be abstracted out and applied across work in general Use these common criteria as the basis for a Definition of Done Assess Technical Debt Identify any rework that needs to be done Identify the reasons why it wasn't done properly the first time Identify what measures can be put in place to stop similar rework from occurring Add these measures to the Definition of Done (DoD) Continually update the DoD In each Sprint Review, identify which (if any) work was rejected or which caused rework to be done, then In each Sprint Retrospective, challenge the DoD for relevance and completeness There isn't a prescribed format for a Definition of Done, but it can be beneficial to use the same as that which is used for Acceptance Criteria, either in whole or in part. This allows a flexible definition based on story type. Alternatively they can be written as simple lists. Here are some examples: Example of a Definition of Done in Acceptance Criteria Format Given that a user story has required a code change When BDD and unit tests have been written for the story and the code change has been reviewed and the code change has been approved by a peer and all BDD and unit tests have been run and no BDD or unit tests have broken (green bar) and the code change has been committed to the repository and QA testing has passed satisfactorily and the Product Owner has approved the change Then the user story will be deployed to production and it will be considered Done. Given that a user story has required the authoring of documentation When the documentation has been reviewed and approved by a peer and the documentation has been approved by the Product Owner Then a new version of the documentation will be committed and the user story will be considered Done. Example of a Definition of Done in List Format Code changes: BDD tests written and pass Unit tests written and pass Code peer reviewed & approved Code committed to repository QA testing done Product Owner signed off Documentation: Documentation has been peer reviewed & approved Documentation approved by Product Owner Version number updated Definitions of Done for IT Infrastructure Support We've seen that having a good Definition of Done is important, although in IT we also need Acceptance Criteria that address the particulars of each User Story. When used in combination they can approach the levels of rigor that have been shown to be possible in manufacturing. Those working in software development can adopt a similar commitment to quality. Now we need to turn our attention to another function within the IT department...Infrastructure Support. Infrastructure support teams are increasingly expected to work in an agile way. As part of an enterprise transformation that does not seem unreasonable. After all, the rest of the organization is highly dependent upon them. Their scope includes such things as deploying new workstations and laptops (possibly across entire sites), installing networks, performing miscellaneous diagnostics and repairs, and maintaining and upgrading local server resources. Clearly they will also need Definitions of Done and Acceptance Criteria if they are to be stakeholders in a joined-up agile enterprise. The question is, how on earth can a meaningful Definition of Done be abstracted across wildly different physical tasks? How can a Definition of Done cover everything from a phone installation to a printer driver upgrade or a memory enhancement, or a firewall configuration to a keyboard replacement? The answer is to focus on the value chain that is represented by each user story. Work is not "released" as such, but rather it is handed over to someone who can derive benefit from it (i.e. the person occupying the user story role). This is the key to understanding "done" in an infrastructure context. If you can identify the parties who derive value, and demonstrably pass that value on to them, then your work is done. Here's an example of a Definition of Done that might be used to close out a support ticket: The receiver of the work has been identified Handover instructions have been completed and given to the receiver The receiver has been notified of the intention to close the ticket, and has not raised an objection A security assessment has been conducted and approved There are three things to point out here. The absence of any reference to a Product Owner. This is because infrastructure teams have to support multiple products, and prioritization of work is traditionally handled not through any sense of ownership of those products, but rather through help-desk triage. It's certainly possible for work to be represented by Product Owners, but it would have to be ownership of the business support function rather than ownership of the actual products being supported. The need to identify and work with the actual receivers of value is still there. The "acceptance by default" position. Receivers typically have little incentive to sign work off as being complete. On the contrary, their incentive is to defer acceptance as long as possible, for potential use as a "banker" in case a requirement for additional unforeseen work transpires. They might hope to ride this new work on an existing ticket instead of having to raise a new one. Receivers can be expected to care about their own support needs, not about the big picture of enterprise delivery. If a Product Owner can be identified - even if it is just the most likely owner of the business support function - then this situation can be improved. A Product Owner can apply leverage for appropriate and timely sign-off, such as by not accepting new work from certain parties while their approval (or justified rejection) of prior work remains outstanding. The elicitation of solid Acceptance Criteria can help the Product Owner immensely. Security implications need to be given careful consideration. The reworking of organizational infrastructure offers great potential for security to be compromised. Approval from Information Security should be obtained for all work, either directly or through authorized agents. One approach is for each team to have a designated "security champion" who provides this function. Conclusion Teams that appear to be healthy can still incur rework and technical debt. A poor understanding of what "done" means often underlies such problems, and this should be one of the first things to be looked at if problems are suspected. Having a meaningful Definition of Done encourages a team's sense of ownership of their own process, and helps instil self-discipline into its members to follow agile best practices. The application of this standard requires finding the right balance between team oversight and trust.

I recently stumbled upon an article Synchronising Multithreaded Integration Tests on Captain Debug's Blog. That post emphasizes the problem of designing integration tests involving class under test running business logic asynchronously. This contrived example was given (I stripped some comments): public class ThreadWrapper { public void doWork() { Thread thread = new Thread() { @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); } private void addDataToDB() { // Dummy Code... try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } }; thread.start(); System.out.println("Off and running..."); } } This is only an example of common pattern where business logic is delegated to some asynchronous job pool we have no control over. Roger Hughes (the author) enumerates few techniques of testing such code, including: arbitrary ("long enough") sleep() in test method to make sure background logic finishes refactoring doWork() so that it accepts CountDownLatch and agrees to notify it when job is done making the method above package private and @VisibleForTesting only "The" solution - refactoring doWork() so that it accepts arbitrary Runnable. In test we can wrap this Runnable (decorator pattern) and wait for inner Runnable to complete Last solution is not bad but it changes the responsibilities of ThreadWrapper significantly. Now it's up to the caller to decide what kind of job should be executed asynchronously while previously ThreadWrapper was encapsulating business logic completely. I am not saying it's a bad design, but it's drastically different from original method. Awaitility Can we write a test without such a massive refactoring? First solution involves handy library called Awaitility. This library is not a silver bullet, it simply evaluates given condition periodically and makes sure it's fulfilled within given time. It's the kind of code you probably wrote once or twice - wrapped in a library with well designed API. So here is our initial approach: import static com.jayway.awaitility.Awaitility.await; import static java.util.concurrent.TimeUnit.SECONDS; //... await().atMost(10, SECONDS).until(recordInserted()); //... private Callable recordInserted() { return new Callable() { @Override public Boolean call() throws Exception { return dataExists(); } }; } I think there is nothing to explain here. dataExists() is simply a boolean method that initially returns false but will eventually return true once the background task (addDataToDB()) is done. In other words we assume that background task introduces some side effect and dataExists() can detect that side effect. BTW I happened to have JDK 8 with Lambda support installed and IntelliJ IDEA gives me this nice tooltip: Suddenly I get this Java 8-compatible alternative suggested: private Callable recordInserted() { return () -> dataExists(); } But there's more: Which transforms my code to: private Callable recordInserted() { return this::dataExists; } this:: prefix means that recordInsterted is a method of current object. Just as well we can say someDao::dataExists. Simply put this syntax turns method into a function object we can pass around (this process is called eta expansion in Scala). By now recordInsterted() method is no longer that needed so I can inline it and remove it completely: await().atMost(10, SECONDS).until(this::dataExists); I am not sure what I love more - the new lambda syntax or how IntelliJ IDEA takes pre-Java 8 code and retrofits it for me automatically (well, it's still a bit experimental, just reported IDEA-106670). I can run this intention in IntelliJ project-wide, Lambda-enabling my whole code base in seconds. Sweet! But back to original problem. Awaitility helps a lot by providing decent API and some handy features. I use it extensively in combination with FluentLenium. But periodically polling for state changes feels a bit like a workaround and still introduces minimal latency. But notice that running and synchronizing on asynchronous tasks is quite common and JDK already provides necessary facilities: Future abstraction! java.util.concurrent.Future To limit the scope of refactoring I will leave the original new Thread() approach for now and use SettableFuture from Guava. It is a Future implementation that allows triggering completion or failure at any time, from any thread (see DeferredResult - asynchronous processing in Spring MVC for more advanced usage). As you can see the changes are quite small: public class ThreadWrapper { public ListenableFuture doWork() { final SettableFuture future = SettableFuture.create(); Thread thread = new Thread() { @Override public void run() { addDataToDB() //... //last instruction future.set(null); } private void addDataToDB() { // Dummy Code... // ... } }; thread.start(); return future; } } doWork() now returns ListenableFuture with lifecycle controlled inside asynchronous task. We use Void but in reality you might want to return some asynchronous result instead. future.set(null) invocation in the end is crucial. It signals that future is fulfilled and all threads waiting for that future will be notified. Once again, in practice you would use e.g. Future and then instead of null we would say future.set(someInteger). Here null is just a placeholder for Void type. How does this help us? Test code can now rely on future completion: final ListenableFuture future = wrapper.doWork(); future.get(10, SECONDS); future.get() blocks until future is done (with timeout), i.e. until we call future.set(...). BTW I use ListenableFuture from Guava but Java 8 introduces equivalent and standard CompletableFuture - I will write about it soon. So, we are getting somewhere. Future is a useful abstraction for waiting and signalling completion of background jobs. But there is also one immense advantage of Future which are not taking, ekhm, advantage from - exception handling and propagation. Future.get() will block until future is complete and return asynchronous result or throw an exception initially thrown from our job. This is really useful for asynchronous tests. Currently if Thread.run() throws an exception it may or may not be logged or visible to us and future will never be completed. With Awaitility it's slightly better - it will timeout without any meaningful reason, which have to be tracked down manually in console/logs. But with minor modification our test is much more verbose: public void run() { try { addDataToDB() //... future.set(null); } catch (Exception e) { future.setException(e); } } If some exception occurs in asynchronous job, it will pop-up and be shown as JUnit/TestNG failure reason. (Listening)ExecutorService That's it. If addDataToDB() throws an exception it will not be lost. Instead our future.get() in test will re-throw that exception for us. Our test won't simply timeout leaving us with no clue what went wrong. Great, but do we really have to create this special SettableFuture instance, can't we just use existing libraries that already give us Future with correct underlying implementation? Of course! By this requires further refactoring: import com.google.common.util.concurrent.ListeningExecutorService; import com.google.common.util.concurrent.MoreExecutors; import java.util.concurrent.Executors; import java.util.concurrent.Future; public class ThreadWrapper { private final ListeningExecutorService executorService = MoreExecutors.listeningDecorator( Executors.newSingleThreadExecutor() ); public ListenableFuture doWork() { Runnable job = new Runnable() { @Override public void run() { //... } }; return executorService.submit(job); } } This is what you've all been waiting for. Don't start new Thread all the time, use thread pool! I actually went one step further by using ListeningExecutorService - an extension to ExecutorService that returns ListenableFuture instances (see why you want that). But the solution doesn't require this, I just spread good practices. As you can see Future instance is now created and managed for us. The test is exactly the same but production code is cleaner and more robust. MoreExecutors.sameThreadExecutor() The final trick I want to show you involves dependency injection. First let's externalize the creation of a thread pool from ThreadWrapper class: private final ListeningExecutorService executorService; public ThreadWrapper() { this(Executors.newSingleThreadExecutor()); } public ThreadWrapper(ExecutorService executorService) { this.executorService = MoreExecutors.listeningDecorator(executorService); } We can now optionally supply custom ExecutorService. This is good for various other reasons, but for us it opens brand new testing opportunity: MoreExecutors.sameThreadExecutor(). This time we modify our test slightly: final ThreadWrapper wrapper = new ThreadWrapper(MoreExecutors.sameThreadExecutor()); wrapper.doWork().get(); See how we pass custom ExecutorService? It's a very special implementation that doesn't really maintain thread pool of any kind. Every time you submit() some task to that "pool" it will be executed in the same thread in a blocking manner. This means that we no longer have asynchronous test, even though the production code wasn't changed that much! wrapper.doWork() will block until "background" job finishes. The extra call to get() is still needed to make sure exceptions are propagated, but is guaranteed to never block (because the job is already done). Using the same thread to execute asynchronous task instead of a thread pool might have an unexpected results if you somehow depend on thread-based properties, e.g. transactions, security, ThreadLocal. However if you use standard ThreadPoolExecutor with CallerRunsPolicy, JDK already behaves this way if thread pool is overflowed. So it's not that unusual. Summary Testing asynchronous code is hard, but you have options. Several options. But one conclusion that strikes me is the side effect of our efforts. We refactored original code in order to make it testable. But the final production code is not only testable, but also much better structured and robust. Surprisingly it's even source-code compatible with previous version as we barely changed return type from void to Future. It seems to be a rule - testable code is often better designed and implemented. Unit test is the first client code using our library. It naturally forces us to to think more about consumers, not the implementation.