Giving Infinitest and Continuous Testing a Try

It has been a week since I started using Infinitest while working on FEST-Assert. I’m very pleased with the results, to the point that I’m getting addicted to it! Infinitest is an interesting tool that brings Continuous Testing to your IDE (Eclipse or IDEA.)

From its website:

Whenever you make a change, Infinitest runs tests for you. It selects tests intelligently, and only runs the ones you need. It reports unit test failures like compiler errors, and provides additional information that helps you write better tests.

I was first introduced to Infinitest by my good friend Rod Coffin, and I was impressed with it the very first time I saw it!

Benefits

Infinitest provides the fastest feedback possible. It will notify us right away if any changes to the code base breaks existing tests. This is done without running the whole test suite, just the tests covering the affected code. That is IMHO, priceless, for the following reasons (in no particular order.)

• Reduced risk of introducing defects: tests are executed immediately after any code changes
• Less broken builds: there is a good chance Infinitest will catch any broken tests before checking-in
• Less manual work: Infinitest runs tests automatically
• (Potentially) less coupled code: if too many tests are running while performing small changes in code, it may be a sign of code that is too tightly coupled

Overall, I’m working faster, with more confidence and having more fun :)

Prerequisites?

Infinitest has been working really well for me. I think this is directly related to the nature of the project I’m working on:

1. It is a small project: only 48 classes and 2,604 NCSS (Non Commenting Source Statements)
2. It has a comprehensive test suite: 1,718 tests with 99.9% code coverage
3. It has unit tests only, no integration or functional tests
4. Tests run fast

As you can see, given this setup, Infinitest runs my tests in the background pretty quickly. I still don’t know if this will be case when using it in a bigger project, with more functional tests.

A more “real” (and hopefully fair) test

After releasing FEST-Assert (hopefully in November,) I’m going to give Infinitest a try with FEST-Swing.

FEST-Swing is a bigger, more complex project:

1. It has 389 classes, and 14,383 NCSS
2. More tests: 3,405
3. Tests are also slower: most of them are functional tests that interact with a Swing UI

Probably FEST-Swing is not a good candidate for continuous testing, since its tests are slower than unit tests. Nothing wrong with it, it is just the nature of the project. I can expect to see some Swing UIs popping up in front of my IDE, taking over the mouse and keyboard, after making some changes in code. I’m very curious to see how this kind of interruption affects my coding “flow.”

Even if it doesn’t work as I expected, I’m pretty sure this exercise can at least help me reorganize the code base, as a way to minimize the number of functional tests executed.

I’ll be posting my experiences with this scenario as well :)

What Infinitest can’t do

I have found one thing Infinitest doesn’t cover: code that uses reflection. I accidentally broke some tests that use reflection to do some funky (but necessary) stuff. To be honest, asking Infinitest to cover code that is using reflection is asking the impossible.

The lesson I learned is:

Some rough edges

Even though I’m a happy Infinitest user, there are things that, in my opinion, can be improved. I’ve been using Eclipse 3.5.1 and Infinitest 5.1.61, and I have a couple of suggestions.

1. Fix: Infinitest executes unrelated tests
When deciding which tests to run, Infinitest is smart, but not smart enough (IMHO.) In my case, I made a small change to the method method read(String) in the class ImageAssert. The only reference to the affected method is the test class ImageAssert_read_Test, which has only two methods. Instead of running ImageAssert_read_Test by itself, Infinitest ran all the tests that have a reference to ImageAssert. Instead of executing only 2 test methods, Infinitest executed 13!

2. Fix: sometimes tests in other projects within a workspace are executed
I haven’t been able to reproduce this issue consistently. For some strange reason, Infinitest tries to run approximately 1,300 tests from the FEST-Swing project, even though there are no dependencies between these two projects. This is particularly painful for me, since most of these tests pop up a Swing UI and take over the mouse. My only option is to kill all Java processes.

3. Fix: remove “quick fix” for failed test
Infinitest reports any broken tests in the “Problems” view in Eclipse, which is fine. The issue here is that the “quick fix” action for a test failure is to “print stacktrace,” which I find pretty much useless. Generally speaking, I don’t think there is a “quick fix” for a test failure. I’d prefer to have this action removed (I’m not sure if it is technically doable though.)

4. Fix: sometimes, changes in tests do not trigger Infinitest
Once again, I haven’t found a consistent way to reproduce this issue.

5. Improvement: display executed tests in its own view
Currently, Infinitest displays executed tests in a tooltip:

I’d like to see the executed tests in a view similar to the “Problems” view: a table where I can select a test, go its source or just execute it again.

6. Improvement: ability to stop Infinitest once it started running tests
This is related with issue #2. I’d like to have ability to stop Infinitest when it is running tests that are either not related to the change I just made or are simply intrusive.

7. Improvement: change icon for test failures
Infinitest shows test failures using the same icon Eclipse uses for compilation errors:

Although it is really no big deal, it can be confusing sometimes. I’d prefer to see a different icon for test failures.

8. Improvement: integrate with Eclipse’s JUnit plug-in
This is related to issue #4. If Infinitest is not triggered by changes I made to a broken test, I’d like to see the broken test removed from the “Problems” view when such test passes when I manually execute it through the JUnit plug-in.

9. Improvement: documentation
Infinitest is so easy to install and use, that there is really no need for documentation. What I’d like to see is documentation about strategies to reorganize both production code and test code, to minimize both the number of tests executed and the time tests are executed in the background.

10. Wish: a graph showing dependencies between code and executed tests
IMHO, it would be nice if Infinitest provided some graphical representation of the relationship between changes being made and the tests being executed, to have a better understanding of these dependencies and help me reorganize my code.

Conclusion

I’m very satisfied with Infinitest. It is an excellent tool that notifies me instantly if any changes I make break existing functionality. In my case, the main benefits are increased productivity and confidence. It is already in my “must have” tool arsenal. Even though I’ve been using it successfully in a small project with a comprehensive test suite, I still have to see how Infinitest performs in a bigger, more complex project that contains a good number of functional tests. Stay tuned!

From http://alexruiz.developerblogs.com