Five quick points about unit tests that will reduce your defect rates.
- Always stop to add that simple unit test. You will be amazed how often this discovers a bug or unexplored corner case.
- Never develop from a main method, find a way to turn it into a test. Main driven development is horrible, they are not part of the continuous development cycle so once finished they get forgotten.
- Make all tests run in the continuous integration environment. A test that does not run is a dark test. Tests that are not continually run are probably broken. Same goes for main method development.
- Expend effort getting the time from commit to all project tests passing tests as small as possible. If I could check in, and see the results 1 second later there is little chance of me holding someone else up. Aim for less than 15 mins with current technology. Parallelism is the key. Separate the build, many small suits that run in parallel. Why not build a cloud just for running parallel test suits in?
- Do as much testing as possible using simple unit tests. Unit tests are cheap to write, run fast and are easier to maintain. Test as much as possible with them, resort to higher level tests only to test integration. Good unit level coverage will also mean the higher levels have less to do.
If you just did the above you would be doing very well indeed.
Some other test related thoughts I had today:
Why would you schedule your test runs on an hourly schedule? Always hook unit tests into the commit. If your unit coverage is high it might be possible to consider the unit tests a pass or fail threshold for a commit.
If you can’t fix a break quickly revert before the breaks pile up. Choose a source control system that makes this easy to do. The threat of a revert means people will go out of their way to test the build before checking in.
Integration tests should also be continuously run. There may be challenges, they are often slower, and more fragile. Break them into batches but not just by functional area. Consider batches split by test categorisation, with buckets like:
Slow running: Schedule infrequent runs.
Fragile but fast: Trigger on commit.
Performance: Run when system is not under load.
The general rule is be inventive, find ways to get them run as frequently as possible.
Finally ask yourself this next time you realise the need for another test, but decide not to write it. Why do I feel its ok not to write this test?
Too hard: Perhaps its time to find better ways to crack the problem. An opportunity to be creative.
Too trivial: It should not take too long to write it then.
Your under pressure to deliver: Local optimization, delivering untested code ultimately slows down delivery.