Martin Fowler [Deactivated] [Suspended]

The loop is the classic way of processing collections, but with the greater adoption of first-class functions in programming languages the collection pipeline is an appealing alternative. In this article I look at refactoring loops to collection pipelines with a series of small examples. I'm publishing this article in installments. This adds an example of refactoring a loop that summarizes flight delay data for each destination airport. A common task in programming is processing a list of objects. Most programmers naturally do this with a loop, as it's one of the basic control structures we learn with our very first programs. But loops aren't the only way to represent list processing, and in recent years more people are making use of another approach, which I call the collection pipeline. This style is often considered to be part of functional programming, but I used it heavily in Smalltalk. As OO languages support lambdas and libraries that make first class functions easier to program with, then collection pipelines become an appealing choice. Refactoring a Simple Loop into a Pipeline I'll start with a simple example of a loop and show the basic way I refactor one into a collection pipeline. Let's imagine we have a list of authors, each of which has the following data structure. class Author... public string Name { get; set; } public string TwitterHandle { get; set;} public string Company { get; set;} This example uses C# Here is the loop. class Author... static public IEnumerable TwitterHandles(IEnumerable authors, string company) { var result = new List (); foreach (Author a in authors) { if (a.Company == company) { var handle = a.TwitterHandle; if (handle != null) result.Add(handle); } } return result; } My first step in refactoring a loop into a collection pipeline is to apply Extract Variable on the loop collection. class Author... static public IEnumerable TwitterHandles(IEnumerable authors, string company) { var result = new List (); var loopStart = authors; foreach (Author a in loopStart) { if (a.Company == company) { var handle = a.TwitterHandle; if (handle != null) result.Add(handle); } } return result; } This variable gives me a starting point for pipeline operations. I don't have a good name for it right now, so I'll use one that makes sense for the moment, expecting to rename it later. I then start looking at bits of behavior in the loop. The first thing I see is a conditional check, I can move this to the pipeline with a . class Author... static public IEnumerable TwitterHandles(IEnumerable authors, string company) { var result = new List (); var loopStart = authors .Where(a => a.Company == company); foreach (Author a in loopStart) { if (a.Company == company) { var handle = a.TwitterHandle; if (handle != null) result.Add(handle); } } return result; } I see the next part of the loop operates on the twitter handle, rather than the author, so I can use a a . class Author... static public IEnumerable TwitterHandles(IEnumerable authors, string company) { var result = new List (); var loopStart = authors .Where(a => a.Company == company) .Select(a => a.TwitterHandle); foreach (string handle in loopStart) { var handle = a.TwitterHandle; if (handle != null) result.Add(handle); } return result; } Next in the loop as another conditional, which again I can move to a filter operation. class Author... static public IEnumerable TwitterHandles(IEnumerable authors, string company) { var result = new List (); var loopStart = authors .Where(a => a.Company == company) .Select(a => a.TwitterHandle) .Where (h => h != null); foreach (string handle in loopStart) { if (handle != null) result.Add(handle); } return result; } All the loop now does is add everything in its loop collection into the result collection, so I can remove the loop and just return the pipeline result. class Author... static public IEnumerable TwitterHandles(IEnumerable authors, string company) { var result = new List (); return authors .Where(a => a.Company == company) .Select(a => a.TwitterHandle) .Where (h => h != null); foreach (string handle in loopStart) { result.Add(handle); } return result; } Here's the final state of the code class Author... static public IEnumerable TwitterHandles(IEnumerable authors, string company) { return authors .Where(a => a.Company == company) .Select(a => a.TwitterHandle) .Where (h => h != null); } What I like about collection pipelines is that I can see the flow of logic as the elements of the list pass through the pipeline. For me it reads very closely to how I'd define the outcome of the loop "take the authors, choose those who have a company, and get their twitter handles removing any null handles". Furthermore, this style of code is familiar even in different languages who have different syntaxes and different names for pipeline operators. Java public List twitterHandles(List authors, String company) { return authors.stream() .filter(a -> a.getCompany().equals(company)) .map(a -> a.getTwitterHandle()) .filter(h -> null != h) .collect(toList()); } Ruby def twitter_handles authors, company authors .select {|a| company == a.company} .map {|a| a.twitter_handle} .reject {|h| h.nil?} end while this matches the other examples, I would replace the final reject with compact Clojure (defn twitter-handles [authors company] (->> authors (filter #(= company (:company %))) (map :twitter-handle) (remove nil?))) F# let twitterHandles (authors : seq, company : string) = authors |> Seq.filter(fun a -> a.Company = company) |> Seq.map(fun a -> a.TwitterHandle) |> Seq.choose (fun h -> h) again, if I wasn't concerned about matching the structure of the other examples I would combine the map and choose into a single step I've found that once I got used to thinking in terms of pipelines I could apply them quickly even in an unfamiliar language. Since the fundamental approach is the same it's relatively easy to translate from even unfamiliar syntax and function names. Refactoring within the Pipeline, and to a Comprehension Once you have some behavior expressed as a pipeline, there are potential refactorings you can do by reordering steps in the pipeline. One such move is that if you have a map followed by a filter, you can usually move the filter before the map like this. class Author... static public IEnumerable TwitterHandles(IEnumerable authors, string company) { return authors .Where(a => a.Company == company) .Where (a => a.TwitterHandle != null) .Select(a => a.TwitterHandle); } When you have two adjacent filters, you can combine them using a conjunction. class Author... static public IEnumerable TwitterHandles(IEnumerable authors, string company) { return authors .Where(a => a.Company == company && a.TwitterHandle != null) .Select(a => a.TwitterHandle); } Once I have a C# collection pipeline in the form of a simple filter and map like this, I can replace it with a Linq expression class Author... static public IEnumerable TwitterHandles(IEnumerable authors, string company) { return from a in authors where a.Company == company && a.TwitterHandle != null select a.TwitterHandle; } I consider Linq expressions to be a form of , and similarly you can do something like this with any language that supports list comprehensions. It's a matter of taste whether you prefer the list comprehension form, or the pipeline form (I prefer pipelines). In general pipelines are more powerful, in that you can't refactor all pipelines into comprehensions.

yagni originally is an acronym that stands for "you aren't gonna need it". it is a mantra from extremeprogramming that's often used generally in agile software teams. it's a statement that some capability we presume our software needs in the future should not be built now because "you aren't gonna need it". yagni is a way to refer to the xp practice of simple design (from the first edition of the white book the second edition refers to the related notion of "incremental design"). like many elements of xp, it's a sharp contrast to elements of the widely held principles of software engineering in the late 90s. at that time there was a big push for careful up-front planning of software development. let's imagine i'm working with a startup in minas tirith selling insurance for the shipping business. their software system is broken into two main components: one for pricing, and one for sales. the dependencies are such that they can't usefully build sales software until the relevant pricing software is completed. at the moment, the team is working on updating the pricing component to add support for risks from storms. they know that in six months time, they will need to also support pricing for piracy risks. since they are currently working on the pricing engine they consider building the presumptive feature for piracy pricing now, since that way the pricing service will be complete before they start working on the sales software. yagni argues against this, it says that since you won't need piracy pricing for six months you shouldn't build it until it's necessary. so if you think it will take two months to build this software, then you shouldn't start for another four months (neglecting any buffer time for schedule risk and updating the sales component). the first argument for yagni is that while we may now think we need this presumptive feature, it's likely that we will be wrong. after all the context of agile methods is an acceptance that we welcome changing requirements. a plan-driven requirements guru might counter argue that this is because we didn't do a good-enough job of our requirements analysis, we should have put more time and effort into it. i counter that by pointing out how difficult and costly it is to figure out your needs in advance, but even if you can, you can still be blind-sided when the gondor navy wipes out the pirates, thus undermining the entire business model. in this case, there's an obvious cost of the presumptive feature - the cost of build : all the effort spent on analyzing, programming, and testing this now useless feature. but let's consider that we were completely correct with our understanding of our needs, and the gondor navy didn't wipe out the pirates. even in this happy case, building the presumptive feature incurs two serious costs. the first cost is the cost of delayed value. by expending our effort on the piracy pricing software we didn't build some other feature. if we'd instead put our energy into building the sales software for weather risks, we could have put a full storm risks feature into production and be generating revenue two months earlier. this cost of delay due to the presumptive feature is two months revenue from storm insurance. the common reason why people build presumptive features is because they think it will be cheaper to build it now rather than build it later. but that cost comparison has to be made at least against the cost of delay, preferably factoring in the probability that you're building an unnecessary feature, for which your odds are at least ⅔. often people don't think through the comparative cost of building now to building later. one approach i use when mentoring developers in this situation is to ask them to imagine any refactoring they would have to do later to introduce the capability when it's needed. often that thought experiment is enough to convince them that it won't be significantly more expensive to add it later. another result from such an imagining is to add something that's easy to do now, adds minimal complexity, yet significantly reduces the later cost. using lookup tables for error messages rather than inline literals are an example that are simple yet make later translations easier to support. reminder, any extensibility point that’s never used isn’t just wasted effort, it’s likely to also get in your way as well -- jeremy miller the cost of delay is one cost that a successful presumptive feature imposes, but another is the cost of carry . the code for the presumptive feature adds some complexity to the software, this complexity makes it harder to modify and debug that software, thus increasing the cost of other features. the extra complexity from having the piracy-pricing feature in the software might add a couple of weeks to how long it takes to build the storm insurance sales component. that two weeks hits two ways: the additional cost to build the feature, plus the additional cost of delay since it look longer to put it into production. we'll incur a cost of carry on every feature built between now and the time the piracy insurance software starts being useful. should we never need the piracy-pricing software, we'll incur a cost of carry on every feature built until we remove the piracy-pricing feature (assuming we do), together with the cost of removing it. so far i've divided presumptive features in two categories: successful and unsuccessful. naturally there's really a spectrum there, and with one point on that spectrum that's worth highlighting: the right feature built wrong. development teams are always learning, both about their users and about their code base. they learn about the tools they're using and these tools go through regular upgrades. they also learn about how their code works together. all this means that you often realize that a feature coded six months ago wasn't done the way you now realize it should be done. in that case you have accumulated technicaldebt and have to consider the cost of repair for that feature or the on-going costs of working around its difficulties. so we end up with three classes of presumptive features, and four kinds of costs that occur when you neglect yagni for them. my insurance example talks about relatively user-visible functionality, but the same argument applies for abstractions to support future flexibility. when building the storm risk calculator, you may consider putting in abstractions and parameterizations now to support piracy and other risks later. yagni says not to do this, because you may not need the other pricing functions, or if you do your current ideas of what abstractions you'll need will not match what you learn when you do actually need them. this doesn't mean to forego all abstractions, but it does mean any abstraction that makes it harder to understand the code for current requirements is presumed guilty. yagni is at its most visible with larger features, but you see it more frequently with small things. recently i wrote some code that allows me to highlight part of a line of code. for this, i allow the highlighted code to be specified using a regular expression. one problem i see with this is that since the whole regular expression is highlighted, i'm unable to deal with the case where i need the regex to match a larger section than what i'd like to highlight. i expect i can solve that by using a group within the regex and letting my code only highlight the group if a group is present. but i haven't needed to use a regex that matches more than what i'm highlighting yet, so i haven't extended my highlighting code to handle this case - and won't until i actually need it. for similar reasons i don't add fields or methods until i'm actually ready to use them. small yagni decisions like this fly under the radar of project planning. as a developer it's easy to spend an hour adding an abstraction that we're sure will soon be needed. yet all the arguments above still apply, and a lot of small yagni decisions add up to significant reductions in complexity to a code base, while speeding up delivery of features that are needed more urgently. now we understand why yagni is important we can dig into a common confusion about yagni. yagni only applies to capabilities built into the software to support a presumptive feature, it does not apply to effort to make the software easier to modify. yagni is only a viable strategy if the code is easy to change, so expending effort on refactoring isn't a violation of yagni because refactoring makes the code more malleable. similar reasoning applies for practices like selftestingcode and continuousdelivery . these are enabling practices for evolutionary design , without them yagni turns from a beneficial practice into a curse. but if you do have a malleable code base, then yagni reinforces that flexibility. yagni has the curious property that it is both enabled by and enables evolutionary design. yagni is not a justification for neglecting the health of your code base. yagni requires (and enables) malleable code. i also argue that yagni only applies when you introduce extra complexity now that you won't take advantage of until later. if you do something for a future need that doesn't actually increase the complexity of the software, then there's no reason to invoke yagni. having said all this, there are times when applying yagni does cause a problem, and you are faced with an expensive change when an earlier change would have been much cheaper. the tricky thing here is that these cases are hard to spot in advance, and much easier to remember than the cases where yagni saved effort . my sense is that yagni-failures are relatively rare and their costs are easily outweighed by when yagni succeeds.

Story points are a common name for sizing stories in agile projects. Combined with XpVelocity they provide a technique to aid planning by providing a forecast of when stories can be completed. When estimating work, a common approach is to estimate in terms staff-hours, such as a programmer saying "this will take me two days to do". Many people in the early days of agile, especially those in the ExtremeProgramming community, found that teams struggled to come up with useful estimates using this approach, even when they applied an approach of IdealTime. We found the most effective way to estimate was to size stories relative to each other, and then use past experience to determine how much could be done in an iteration. [1] To determine the points for a story, we compare rough relative sizes. If we are estimating the "fibble the foobar" story, we look for a story of similar size that we've already estimated. We sense it's about the same size as "flipping the synergy bit". Then we look at the story point score for "flipping the synergy bit" and score the "fibble the foobar" the same amount. A team using story points uses a small range of story points to work with. Common examples might be 1,2,4,8 or 1,2,3,5,8 [2]. Often the top number in the series represents "too big" and should be broken down further. [3] Allocating story points should be rapid activity. Discussion should only break out when people have contrasting views on the estimate, in which case its useful to have a discussion as it usually means that something about the story isn't clear. Using a ThrownEstimate is a good technique to move things along quickly. To form a plan with time, you use XpVelocity. Some teams don't like using story points, preferring instead to use StoryCounting. I don't have a preference between the two - both seem to work equally well so it's up to the team to try out and go with whichever suits them best. Further Reading The ThoughtWorks ebook on estimation provides includes a good Q&A on story points. Kent and I discussed them in more depth in the tasteful green book. Most books that talk about planning and estimation in an agile context discuss story points in more detail. Notes 1: "Story Points" is the most common name that I hear these days, but various terms have been used over the years, often with whimsical names that emphasized their arbitrary nature. I particularly like Joseph Pelrine's gummi bears and Josh Kerievsky's: NUTs (Nebulous Units of Time). 2: This is a Fibonacci sequence 3: Using the top number as too big is saying that a story sized at '8' really means '8 or more'. If you do this beware of using this top number when making forecasts of things like completion time, since '8' can turn into all sorts of numbers when it finally gets broken down. It's usually better to explicitly say its too big to be estimated rather than use a false marker number.

it can be finicky business to keep a production server running. you have to ensure the operating system and any other dependent software is properly patched to keep it up to date. hosted applications need to be upgraded regularly. configuration changes are regularly needed to tweak the environment so that it runs efficiently and communicates properly with other systems. this requires some mix of command-line invocations, jumping between gui screens, and editing text files. the result is a unique snowflake - good for a ski resort, bad for a data center. the first problem with a snowflake server is that it's difficult to reproduce. should your hardware start having problems, this means that it's difficult to fire up another server to support the same functions. if you need to run a cluster, you get difficulties keeping all of the instances of the cluster in sync. you can't easily mirror your production environment for testing. when you get production faults, you can't investigate them by reproducing the transaction execution in a development environment. [1] making disk images of the snowflake can help to some extent with this. but such images easily gather cruft as unnecessary elements of the configuration, not to mention mistakes, perpetuate. the true fragility of snowflakes, however, comes when you need to change them. snowflakes soon become hard to understand and modify. upgrades of one bit software cause unpredictable knock-on effects. you're not sure what parts of the configuration are important, or just the way it came out of the box many years ago. their fragility leads to long, stressful bouts of debugging. you need manual processes and documentation to support any audit requirements. this is one reason why you often see important software running on ancient operating systems. a good way to avoid snowflakes is to hold the entire operating configuration of the server in some form of automated recipe. two tools that have become very popular for this recently are puppet and chef . both allow you to define the operating environment in a form of domainspecificlanguage , and easily apply it to a given system. the point of using a recipe is not just that you can easily rebuild the server (which you could also do with imaging) but you can also easily understand its configuration and thus modify it more easily. furthermore, since this configuration is a text file, you can keep it in version control with all the advantages that brings. if you disable any direct shell access to the server and force all configuration changes to be applied by running the recipe from version control, you have an excellent audit mechanism that ensures every change to the environment is logged. this approach can be very welcome in regulated environments. application deployment should follow a similar approach: fully automated, all changes in version control. by avoiding snowflakes, it's much easier to have test environments be true clones of production, reducing production bugs caused by configuration differences. a good way of ensuring you are avoiding snowflakes is to use phoenixservers . using version-controlled recipes to define server configurations is an important part of continuous delivery . further reading the visible ops handbook is the pioneering book that talked about the dangers of snowflakes and how to avoid them. continuous delivery talks about how this approach is a necessary part of a sane build and delivery process. true artists, however, prefer snowflakes . 1: another metaphor i've heard for this is that you should treat your servers like cattle and not like pets. although i confess i find it odd when this metaphor is used by my vegetarian colleagues.

while i was at the qcon conference in london a couple of months ago, it seemed that every talk included some snarky remarks about object/relational mapping (orm) tools. i guess i should read the conference emails sent to speakers more carefully, doubtless there was something in there telling us all to heap scorn upon orms at least once every 45 minutes. but as you can tell, i want to push back a bit against this orm hate - because i think a lot of it is unwarranted. the charges against them can be summarized in that they are complex, and provide only a leaky abstraction over a relational data store. their complexity implies a grueling learning curve and often systems using an orm perform badly - often due to naive interactions with the underlying database. there is a lot of truth to these charges, but such charges miss a vital piece of context. the object/relational mapping problem is hard . essentially what you are doing is synchronizing between two quite different representations of data, one in the relational database, and the other in-memory. although this is usually referred to as object-relational mapping, there is really nothing to do with objects here. by rights it should be referred to as in-memory/relational mapping problem, because it's true of mapping rdbmss to any in-memory data structure. in-memory data structures offer much more flexibility than relational models, so to program effectively most people want to use the more varied in-memory structures and thus are faced with mapping that back to relations for the database. the mapping is further complicated because you can make changes on either side that have to be mapped to the other. more complication arrives since you can have multiple people accessing and modifying the database simultaneously. the orm has to handle this concurrency because you can't just rely on transactions- in most cases, you can't hold transactions open while you fiddle with the data in-memory. i think that if you if you're going to dump on something in the way many people do about orms, you have to state the alternative. what do you do instead of an orm? the cheap shots i usually hear ignore this, because this is where it gets messy. basically it boils down to two strategies, solve the problem differently (and better), or avoid the problem. both of these have significant flaws. a better solution listening to some critics, you'd think that the best thing for a modern software developer to do is roll their own orm. the implication is that tools like hibernate and active record have just become bloatware, so you should come up with your own lightweight alternative. now i've spent many an hour griping at bloatware, but orms really don't fit the bill - and i say this with bitter memory. for much of the 90's i saw project after project deal with the object/relational mapping problem by writing their own framework - it was always much tougher than people imagined. usually you'd get enough early success to commit deeply to the framework and only after a while did you realize you were in a quagmire - this is where i sympathize greatly with ted neward's famous quote that object-relational mapping is the vietnam of computer science [1] . the widely available open source orms (such as ibatis, hibernate, and active record) did a great deal to remove this problem [2] . certainly they are not trivial tools to use, as i said the underlying problem is hard, but you don't have to deal with the full experience of writing that stuff (the horror, the horror). however much you may hate using an orm, take my word for it - you're better off. i've often felt that much of the frustration with orms is about inflated expectations. many people treat the relational database "like a crazy aunt who's shut up in an attic and whom nobody wants to talk about" [3] . in this world-view they just want to deal with in-memory data-structures and let the orm deal with the database. this way of thinking can work for small applications and loads, but it soon falls apart once the going gets tough. essentially the orm can handle about 80-90% of the mapping problems, but that last chunk always needs careful work by somebody who really understands how a relational database works. this is where the criticism comes that orm is a leaky abstraction. this is true, but isn't necessarily a reason to avoid them. mapping to a relational database involves lots of repetitive, boiler-plate code. a framework that allows me to avoid 80% of that is worthwhile even if it is only 80%. the problem is in me for pretending it's 100% when it isn't. david heinemeier hansson, of active record fame, has always argued that if you are writing an application backed by a relational database you should damn well know how a relational database works. active record is designed with that in mind, it takes care of boring stuff, but provides manholes so you can get down with the sql when you have to. that's a far better approach to thinking about the role an orm should play. there's a consequence to this more limited expectation of what an orm should do. i often hear people complain that they are forced to compromise their object model to make it more relational in order to please the orm. actually i think this is an inevitable consequence of using a relational database - you either have to make your in-memory model more relational, or you complicate your mapping code. i think it's perfectly reasonable to have a more relational domain model in order to simplify your object-relational mapping. that doesn't mean you should always follow the relational model exactly, but it does mean that you take into account the mapping complexity as part of your domain model design. so am i saying that you should always use an existing orm rather than doing something yourself? well i've learned to always avoid saying "always". one exception that comes to mind is when you're only reading from the database. orms are complex because they have to handle a bi-directional mapping. a uni-directional problem is much easier to work with, particularly if your needs aren't too complex and you are comfortable with sql. this is one of the arguments for cqrs . so most of the time the mapping is a complicated problem, and you're better off using an admittedly complicated tool than starting a land war in asia. but then there is the second alternative i mentioned earlier - can you avoid the problem? avoiding the problem to avoid the mapping problem you have two alternatives. either you use the relational model in memory, or you don't use it in the database. to use a relational model in memory basically means programming in terms of relations, right the way through your application. in many ways this is what the 90's crud tools gave you. they work very well for applications where you're just pushing data to the screen and back, or for applications where your logic is well expressed in terms of sql queries. some problems are well suited for this approach, so if you can do this, you should. but its flaw is that often you can't. when it comes to not using relational databases on the disk, there rises a whole bunch of new champions and old memories. in the 90's many of us (yes including me) thought that object databases would solve the problem by eliminating relations on the disk. we all know how that worked out. but there is now the new crew of nosql databases - will these allow us to finesse the orm quagmire and allow us to shock-and-awe our data storage? as you might have gathered , i think nosql is technology to be taken very seriously. if you have an application problem that maps well to a nosql data model - such as aggregates or graphs - then you can avoid the nastiness of mapping completely. indeed this is often a reason i've heard teams go with a nosql solution. this is, i think, a viable route to go - hence my interest in increasing our understanding of nosql systems. but even so it only works when the fit between the application model and the nosql data model is good. not all problems are technically suitable for a nosql database. and of course there are many situations where you're stuck with a relational model anyway. maybe it's a corporate standard that you can't jump over, maybe you can't persuade your colleagues to accept the risks of an immature technology. in this case you can't avoid the mapping problem. so orms help us deal with a very real problem for most enterprise applications. it's true they are often misused, and sometimes the underlying problem can be avoided. they aren't pretty tools, but then the problem they tackle isn't exactly cuddly either. i think they deserve a little more respect and a lot more understanding. 1: i have to confess a deep sense of conflict with the vietnam analogy. at one level it seems like a case of the pathetic overblowing of software development's problems to compare a tricky technology to war. nasty the programming may be, but you're still in a relatively comfy chair, usually with air conditioning, and bug-hunting doesn't involve bullets coming at you. but on another level, the phrase certainly resonates with the feeling of being sucked into a quagmire. 2: there were also commercial orms, such as toplink and kodo. but the approachability of open source tools meant they became dominant. 3: i like this phrase so much i feel compelled to subject it to re-use.

An automated regression suite can play a vital role on a software project, valuable both for reducing defects in production and essential for evolutionary design. In talking with development teams I've often heard about the problem of non-deterministic tests - tests that sometimes pass and sometimes fail. Left uncontrolled, non-deterministic tests can completely destroy the value of an automated regression suite. In this article I outline how to deal with non-deterministic tests. Initially quarantine helps to reduce their damage to other tests, but you still have to fix them soon. Therefore I discuss treatments for the common causes for non-determinism: lack of isolation, asynchronous behavior, remote services, time, and resource leaks. I've enjoyed watching ThoughtWorks tackle many difficult enterprise applications, bringing successful deliveries to many clients who have rarely seen success. Our experiences have been a great demonstration that agile methods, deeply controversial and distrusted when we wrote the manifesto a decade ago, can be used successfully. There are many flavors of agile development out there, but in what we do there is a central role for automated testing. Automated testing was a core approach to Extreme Programming from the beginning, and that philosophy has been the biggest inspiration to our agile work. So we've gained a lot of experience in using automated testing as a core part of software development. Automated testing can look easy when presented in a text book. And indeed the basic ideas are really quite simple. But in the pressure-cooker of a delivery project, trials come up that are often not given much attention in texts. As I know too well, authors have a habit of skimming over many details in order to get a core point across. In my conversations with our delivery teams, one recurring problem that we've run into is tests which have become unreliable, so unreliable that people don't pay much attention to whether they pass or fail. A primary cause of this unreliability is that some tests have become non-deterministic. A test is non-deterministic when it passes sometimes and fails sometimes, without any noticeable change in the code, tests, or environment. Such tests fail, then you re-run them and they pass. Test failures for such tests are seemingly random. Non-determinism can plague any kind of test, but it's particularly prone to affect tests with a broad scope, such as acceptance or functional tests. Why non-deterministic tests are a problem Non-deterministic tests have two problems, firstly they are useless, secondly they are a virulent infection that can completely ruin your entire test suite. As a result they need to be dealt with as soon as you can, before your entire deployment pipeline is compromised. I'll start with expanding on their uselessness. The primary benefit of having automated tests is that they provide bug detection mechanism by acting as regression tests[1]. When a regression test goes red, you know you've got an immediate problem, often because a bug has crept into the system without you realizing. Having such a bug detector has huge benefits. Most obviously it means that you can find and fix bugs just after they are introduced. Not just does this give you the warm fuzzies because you kill bugs quickly, it also makes it easier to remove them since you know the bug got in with the last set of changes that are fresh in your mind. As a result you know where to look for the bug, which is more than half the battle in squashing it. The second level of benefit is that as you gain confidence in your bug detector, you gain the courage to make big changes knowing that when you goof, the bug detector will go off and you can fix the mistake quickly. [2] Without this teams are frightened to make the changes code needs in order to be kept clean, which leads to a rotting of the code base and plummeting development speed. The trouble with non-deterministic tests is that when they go red, you have no idea whether its due to a bug, or just part of the non-deterministic behavior. Usually with these tests a non-deterministic failure is relatively common, so you end up shrugging your shoulders when these tests go red. Once you start ignoring a regression test failure, then that test is useless and you might as well throw it away. Indeed you really ought to throw a non-deterministic test away, since if you don't it has an infectious quality. If you have a suite of 100 tests with 10 non-deterministic tests in them, than that suite will often fail. Initially people will look at the failure report and notice that the failures are in non-deterministic tests, but soon they'll lose the discipline to do that. Once that discipline is lost, then a failure in the healthy deterministic tests will get ignored too. At that point you've lots the whole game and might as well get rid of all the tests. Quarantine My principal aim in this article is to outline common cases of non-deterministic tests and how to eliminate the non-determinism. But before I get there I offer one piece of essential advice: quarantine your non-deterministic tests. If you have non-deterministic tests keep them in a different test suite to your healthy tests. That way you'll you can continue to pay attention to what's going on with your healthy tests and get good feedback from them. Place any non-deterministic test in a quarantined area. (But fix quarantined tests quickly.) Then the question is what to do with the quarantined test suites. They are useless as regression tests, but they do have a future as work items for cleaning up. You should not abandon such tests, since any tests you have in quarantine are not helping you with your regression coverage. A danger here is that tests keep getting thrown into quarantine and forgotten, which means your bug detection system is eroding. As a result it's worthwhile to have a mechanism that ensures that tests don't stay in quarantine too long. I've come across various ways to do this. One is a simple numeric limit: e.g. only allow 8 tests in quarantine. Once you hit the limit you must spend time to clear all the tests out. This has the advantage of batching up your test-cleaning if that's how you like to do things. Another route is to put a time limit on how long a test may be in quarantine, such as no longer than a week. The general approach with quarantine is to take the quarantined tests out of the main deployment pipeline so that you still get your regular build process. However a good team can be more aggressive. Our Mingle team puts its quarantine suite into the deployment pipeline one stage after its healthy tests. That way it can get the feedback from the healthy tests but is also forced to ensure that it sorts out the quarantined tests quickly. [3] Lack of Isolation In order to get tests to run reliably, you must have clear control over the environment in which they run, so you have a well-known state at the beginning of the test. If one test creates some data in the database and leaves it lying around, it can corrupt the run of another test which may rely on a different database state. Therefore I find it's really important to focus on keeping tests isolated. Properly isolated tests can be run in any sequence. As you get to larger operational scope of functional tests, it gets progressively harder to keep tests isolated. When you are tracking down a non-determinism, lack of isolation is a common and frustrating cause. Keep your tests isolated from each other, so that execution of one test will not affect any others. There are a couple of ways to get isolation - either always rebuild your starting state from scratch, or ensure that each test cleans up properly after itself. In general I prefer the former, as it's often easier - and in particular easier to find the source of a problem. If a test fails because it didn't build up the initial state properly, then it's easy to see which test contains the bug. With clean-up, however, one test will contain the bug, but another test will fail - so it's hard to find the real problem. Starting from a blank state is usually easy with unit tests, but can be much harder with functional tests [4] - particularly if you have a lot of data in a database that needs to be there. Rebuilding the database each time can add a lot of time to test runs, so that argues for switching to a clean-up strategy. One trick that's handy when you're using databases, is to conduct your tests inside a transaction, and then to rollback the transaction at the end of the test. That way the transaction manager cleans up for you, reducing the chance of errors[5]. Another approach is to do a single build of a mostly-immutable starting fixture before running a group of tests. Then ensure that the tests don't change that initial state (or if they do, they reverse the changes in tear-down). This tactic is more error-prone than rebuilding the fixture for each test, but it may be worthwhile iff it takes too long to build the fixture each time. Although databases are a common cause for isolation problems, there are plenty of times you can get these in-memory too. In particular be aware with static data and singletons. A good example for this kind of problem is contextual environment, such as the currently logged in user. If you have an explicit tear-down in a test, be wary of exceptions that occur during the tear-down. If this happens the test can pass, but cause isolation failures for subsequent tests. So ensure that if you do get a problem in a tear-down, it makes a loud noise. Some people prefer to put less emphasis on isolation and more on defining clear dependencies to force tests to run in a specified order. I prefer isolation because it gives you more flexibility in running subsets of tests and parallelizing tests. Asynchronous Behavior Asynchrony is a boon that allows you to keep software responsive while taking on long term tasks. Ajax calls allow a browser to stay responsive while going back to the server for more data, asynchronous message allow a server process to communicate with other system without being tied to their laggardly latency. But in testing, asynchrony can be curse. The common mistake here is to throw in a sleep: //pseudo-code makeAsyncCall; sleep(aWhile); readResponse; This can bite you two ways. First off you'll want to set the sleep time to long enough that it gives plenty of time to get the response. But that means that you'll spend a lot of time idly waiting for the response, thus slowing down your tests. The second bite is that, however long you sleep, sometimes it won't be enough. There will be some change in environment that will cause you to exceed the sleep - and you'll get false failure. As a result I strongly urge you to never use bare sleeps like this. Never use bare sleeps to wait for asynchonous responses: use a callback or polling. There are basically two tactics you can do for testing an asynchronous response. The first is for the asynchronous service to take a callback which it can call when done. This is the best since it means you'll never have to wait any longer than you need to [6]. The biggest problem with this is that the environment needs to be able to do this and then the service provider needs to do it. This is one of the advantages of having the development team integrated with testing - if they can provide a callback then they will. The second option is to poll on the answer. This is more than just looking once, but looking regularly, something like this //pseudo-code makeAsyncCall startTime = Time.now; while(! responseReceived) { if (Time.now - startTime > waitLimit) throw new TestTimeoutException; sleep (pollingInterval); } readResponse The point of this approach is that you can set the pollingInterval to a pretty small value, and know that that's the maximum amount of dead time you'll lose to waiting for a response. This means you can set the waitLimit very high, which minimizes the chance of hitting it unless something serious has gone wrong. [7] Make sure you use a clear exception class that indicates this is a test timeout that's failing. This will help make it clear what's gone wrong should it happen, and perhaps allow a more sophisticated test harness to take account of this information in its display. The time values, in particular the waitLimit, should never be literal values. Make sure they are always values that can be easily set in bulk, either by using constants or set through the runtime environment. That way if you need to tweak them (and you will) you can tweak them all quickly. All this advice is handy for async calls where you expect a response from the provider, but how about those where there is no response. These are calls where we invoke a command on something and expect it to happen without any acknowledgment. This is the trickiest case since you can test for your expected response, but there's nothing to do to detect a failure other than timing-out. If the provider is something you're building you can handle this by ensuring the provider implements some way of indicating that it's done - essentially some form of callback. Even if only the testing code uses it, it's worth it - although often you'll find this kind of functionality is valuable for other purposes too[8]. If the provider is someone else's work, you can try persuasion, but otherwise may be stuck. Although this is also a case when using Test Doubles for remote services is worthwhile (which I'll discuss more in the next section). If you have a general failure in something asynchronous, such that it's not responding at all, then you'll always be waiting for timeouts and your test suite will take a long time to fail. To combat this it's a good idea to use a smoke test to check that the asynchronous service is responding at all and stop the test run right away if it isn't. Gerard Meszaros's book, xUnit Test Patterns, contains lots of good patterns for constructing tests. You can also often side-step the asynchrony completely. Gerard Meszaros's Humble Object pattern says that whenever you have some logic that's in a hard-to-test environment, you should isolate the logic you need to test from that environment. In this case it means put most of the logic you need to test in a place where you can test it synchronously. The asynchronous behavior should be as minimal (humble) as possible, that way you don't need that much testing of it. Remote Services Sometimes I'm asked if ThoughtWorks does any integration work, which I find somewhat amusing since there's hardly any project we do that doesn't involve a fair bit of integration. By their nature, enterprise applications involve a great deal of combining data from different systems. These systems are maintained by other teams operating to their own schedules, teams that often use a very different software philosophy to our heavily test-driven agile approach. Testing with such remote systems brings a number of problems, and non-determinism is high on the list. Often remote systems don't have test system we can call, which means hitting a live system. If there is a test system, it may not be stable enough to provide deterministic responses. In this situation it's vital to ensure determinism, so it's time to reach for a Test Double - a component that looks like the remote service, but is really just a pretend version that mimics the remote system's behavior. The double needs to be setup so that provides the right kind of response in interaction with our system, but in a way we control. In this manner we can ensure determinism. Using a double has a downside, in particular when we are testing across a broad scope. How can we be sure that the double behaves in the same way that remote system does? We can tackle this again using tests, a form of test that I call Integration Contract Tests. These run the same interaction with the remote system and the double, and check that the two match. In this case 'match' may not mean coming up with the same result (due to the non-determinisms), but results that share the same essential structure. Integration Contract Tests need to be run frequently, but not part of our system's deployment pipeline. Periodic running based on the rate of the change of the remote system is usually best. For writing these kinds of test doubles, I'm a big fan of Self Initializing Fakes - since these are very simple to manage. Some people are firmly against using Test Doubles in functional tests, believing that you must test with real connection in order to ensure end-to-end behavior. While I sympathize with their argument, automated tests are useless if they are non-deterministic. So any advantage you gain by talking to the real system is overwhelmed by the need to stamp out non-determinism[9]. Time Few things are more non-deterministic than a call to the system clock. Each time you call it, you get a new result, and any tests that depend on it can thus change. Ask for all the todos due in the next hour, and you regularly get a different answer[10]. The most important thing here is to ensure that you always wrap the system clock with routines that can be replaced with a seeded value for testing. A clock stub can be set to particular time and frozen at that time, allowing your tests to have complete control over its movements. That way you can synchronize your test data to the values in the seeded clock.[11][12] Always wrap the system clock, so it can be easily substituted for testing. One thing to watch with this, is that eventually your test data might start having problems because it's too old, and you get conflicts with other time based factors in your application. In this case you can move the data, and your clock seeds to new values. When you do this, ensure that this is the only thing you do. That way you can be sure that any tests that fail are due to time-movement in the test data. Another area where time can be a problem is when you rely on other behaviors from the clock. I once saw a system that generated random keys based on clock values. This systems started failing when it was moved to a faster machine that could allocate multiple ids within a single clock tick.[13] I've heard so many problems due to direct calls to the system clock that I'd argue for finding a way to use code analysis to detect any direct calls to the system clock and failing the build right there. Even a simple regex check might save you a frustrating debugging session after a call at an ungodly hour. Resource Leaks If your application has some kind of resource leak, this will lead to random tests failing, since it's just which test causes the resource leak to go over a limit that gets the failure. This case is awkward because any test can fail intermittently due to this problem. If it isn't a case of one test being non-deterministic then resource leaks are a good candidate to investigate. By resource leak, I mean any resource that the application has to manage by acquiring and releasing. In non-memory-managed environments, the obvious example is memory. Memory-management did much to remove this problem, but other resources still need to be managed, such as database connections. Usually the best way to handle these kind of resources is through a Resource Pool. If you do this then a good tactic is to configure the pool to a size of 1 and make it throw an exception should it get a request for a resource when it has none left to give. That way the first test to request a resource after the leak will fail - which makes it a lot easier to find the problem test. This idea of limiting resource pool sizes, is about increasing constraints to make errors more likely to crop up in tests. This is good because we want errors to show in tests so we can fix them before they manifest themselves in production. This principle can be used in other ways too. One story I heard was of a system which generated randomly named temporary files, didn't clean them up properly, and crashed on a collision. This kind of bug is very hard to find, but one way to manifest it is to stub the randomizer for testing so it always returns the same value. That way you can surface the problem more quickly.

The emergence of version control systems makes both promiscuous and continuous integration merging techniques more attractive. Which is better?