How to Deal with Slow Unit Tests with Visual Studio Test Runner

one of the most dreadful problem of unit testing is slow testing. if your whole suite of tests runs in 10 minutes, it is normal for developers not to run the whole suite at each build. one of the most common question is

how can i deal with slow unit tests?

here is my actual scenario: in a project i’m working in, we have some multilingual full text search done in elastic search and we have a battery of unit tests that verify that searches work as expected. since each test deletes all documents, insert a bunch of new documents and finally commits lucene index, execution times is high compared to the rest of tests. each test need almost 2 seconds to run on my workstation, where i have really fast ssd and plenty of ram.

this kind of tests cannot be run in memory or with some fancy trick to make then run quickly. actually we have about 30 tests that executes in less than one seconds, and another 13 tests that runs in about 23 seconds, this is clearly unacceptable . after few hours of work, we already reached the point where running the whole suite becomes annoying.

the solution

this is a real common problem and it is quite simple to fix. first of all visual studio test runner actually tells you execution time for each unit test, so you can immediately spot slow tests. when you identify slow tests you can mark them with a specific category, i use slowtest

1
2
3
4
[testfixture]
[category("elasticsearch")]
[category("slowtest")]
public class essearcherfixture : basetestfixturewithhelpers

since i know in advance that this test are slow i immediately mark the entire class with the attribute slowtest. if you have no idea what of your tests are slow, i suggest grouping test by duration in visual studio test runner.

figure 1: group tests by duration

the result is interesting, because visual studio consider every test that needs more than one second to be slow. i tend to agree with this distinction.

figure 2: test are now grouped by duration

this permits you to immediately spot slow tests, so you can add the category slowtest to them. if you keep your unit tests organized and with a good usage of categories, you can simply ask vs test runner to exclude slow test with filter – traits:”slowtest”

figure 3: thanks to filtering i can now execute continuously only test that are not slow.

i suggest you to do a periodic check to verify that every developers is using the slowtest category wisely, just group by duration, filters out the slowtest and you should not have no tests that are marked slow.

figure 4: removing the slowtest category and grouping by duration should list no slow test.

the nice part is that i’m using nunit, because visual studio test runner supports many unit tests frameworks thanks to the concepts of test adapters.

if you keep your tests well organized you will gain maximum benefit from them :).