Tweaking xUnit

one of the interesting challenges that we have with ravendb is the number and duration of our tests.

in particular, we current have over three thousands tests, and they take hours to run. we are doing a lot of stuff there “let us insert million docs, write a map/reduce index, query on that, then do a mass update, see what happens”, etc. we are also doing a lot of stuff that really can’t be emulated easily. if i’m testing replication for a non existent target, i need to check that actual behavior, etc. oh, and we’re probably doing silly stuff in there, too.

in order to try to increase our feedback cycle times, i made some modifications to xunit. it is now going to record the test duration of the tests, the results look like that:

you can see that troy is taking too long. in fact, there is a bug that those tests currently expose that result in a timeout exception, that is why they take so long.

but this is just to demonstrate the issue. the real power here is that we also use this when decided how to run the tests. we are simply sorting them by how long they took to run. if we don’t have a record for that, we’ll give them a run time of –1.

this has a bunch of interesting implications:

• the faster tests are going to run first. that means that we’ll have earlier feedback if we broke something.
• the new tests (haven’t had chance to run ever) will run first, those are were the problems are more likely anyway.
• we only run report this for passing tests, that means that we are going to run failed tests first as well.

in addition to that, this will also give us better feedback on what are slow tests are. so we can actually give them some attention and see if they are really required to be slow or they can be fixed.

hopefully, we can find a lot of the tests that are long, and just split them off into a separate test project, to be run at a later time.

the important thing is, now we have the information to handle this.