For a Fistful of Dollars: Quantifying the Benefits of TDD
According to a recent scientific study, using TDD increases development (coding) time by 15-30%, but results in 40-90% fewer defects. This study was done with 4 different development teams, from IBM (1 team) and Microsoft (3 teams), whose development practices are nothing if not pragmatic. This actually confirms what any TDD practitioner will tell you: you spend a bit more time writing tests up front, but the quality of the resulting code is largely superior.
This is not the only study in this area. There are many others. Uncle Bob Martin sums up quite a few in one of his blog entries on the subject. And Misko Hevery, for example, recently spent two weeks tracking the time spent writing tests, and came to the conclusion that his team spends around 10% of their time writing tests (thanks to Alejandro Cuesta for pointing out this study).
It is interesting to note that the figure of 15-30% longer during the coding phase. Then, the testers found 40-90% fewer bugs. That's 40-90% fewer bugs that need fixing. Now, these bugs that need fixing were found in the functional testing phase. Exact figures will vary, but it is frequently observed that bugs found here will take at least 10 times longer to fix than had they been found during development. When you factor that in, investing 15-30% extra time to write unit tests makes a lot of sense.
A concrete example: the other day, I heard about an internal study at one company, using a more "traditional" (aka "waterfall") approach. The time spent in each phase was: 5 months analysis, 2 months coding, and 7-8 months debugging. So 4 times as much time was spent debugging the code as was spent coding it in the first place! What if you could reduce that 7-8 months by 40-90% (so saving between 3 and 7 months), at the cost of spending another couple of weeks of unit tests using a TDD approach?
And this doesn't even take into account the most important benefits of TDD and BDD. Remember, TDD is not a testing technique, it's a design strategy. TDD code is almost always of much higher quality, more flexible and easier to maintain than the same code developed using more traditional methods. And TDD has an important effect on developers, in addition to encouraging clean design practices: it helps them focus on the requirements. Another internal study showed that one large waterfall-driven project implemented less than 50% of the documented requirements, and that more than half of the implemented requirements were never actually used by the users.
Other interesting lessons reported by the study (which in fact reflect industry best practices in the TDD space), where:
- It is important to start TDD from the beginning of projects
- Write a new unit test every time you need to fix a bug
- Use Continuous Integration
- Encourage fast unit tests
These principles are well-known to TDD practitioners, but always it's nice to seem them confirmed by research.
However TDD is not all roses. It's hard to learn for a lot of developers - unlearning old habits often is. Less experienced TDDers often forget the refactoring part of the test-code-refactor cycle, which means that the TDD process can't do it's job as a design practice. And everyone needs to be on the boat, including management. It's hard (not impossible, just hard) to do TDD by yourself when everyone else is doing things the traditional way.
Nor is TDD appropriate for all development situations, or a replacement for high-level architecture or design. Fine-tuning user Interfaces can be tricky, for example. Uncle Bob Martin has written an excellent blog entry on this topic.
So TDD is not a silver bullet. But it is a confirmed engineering practice that works very well indeed, for a great many situations. The overhead is minimal, the gains huge. So if you're not doing it, you should give it a try. A great place to start is Lasse Koskela's excellent book Test Driven: TDD and Acceptance TDD for Java Developers. Or, in the .NET space, Test-Driven Development in Microsoft .NET, by James Newkirk and Alexei Vorontsov. There are also many excellent TDD training courses around, including the new "Testing and TDD for Java Developers" course which I will be kicking off in New Zealand and Australia over the coming months.