Unit Testing: The Myth of Complete Isolation

Time and time again, when people talk about unit testing, they mention the fragility of unit tests as some inherent disadvantage of the technique. I disagree. I believe this is just bad unit testing and that the people who hold and spread such opinions are simply victims of what I call The Myth of Complete Isolation.

Fragile Unit Tests

Whether these people believe in it or they are just copy-pasting through their programming career, most of those who experience the pains of fragile unit tests practice excessive isolation of their units under test. They do that by mocking out any single thing their object might be cooperating with.

Let’s consider an imaginary Order class:

public class Order {
    private List<OrderItem> items;

    // c-tor

    public BigDecimal getTotalPrice() {
        return items.stream()
                .map(item -> item.getPrice().multiply(item.getQuantity()))
                .reduce(BigDecimal.ZERO, BigDecimal::add);
    }
}

class OrderItem {
    private BigDecimal price;
    private BigDecimal quantity;

    // c-tor, getters
}

If we were to test this class in the mock-everything style, it would look somewhat like this:

public class FragileOrderTest {

    @Test
    public void shouldSumItemPrices() throws Exception {
        OrderItem item1 = orderItem(2, 3);
        OrderItem item2 = orderItem(4, 5);
        Order order = new Order(asList(item1, item2));
        assertEquals(BigDecimal.valueOf(26), order.getTotalPrice());
    }

    private OrderItem orderItem(int price, int quantity) {
        OrderItem item = mock(OrderItem.class);
        when(item.getPrice()).thenReturn(BigDecimal.valueOf(price));
        when(item.getQuantity()).thenReturn(BigDecimal.valueOf(quantity));
        return item;
    }
}

Of course, I made the example as small as possible while still making the point, so I don’t waste your valuable time. In reality, these mocking tests usually look much worse than that and contain ridiculous practices like verifying a number of interactions with every single method (sic!).

Now, with code like the one above, if the communication between Order and OrderItem changed for whatever reason, I would have to change every single test related to them. Just consider this simple, obvious refactoring that should be screaming right at your face from the very first code example:

public class Order {
    // same as before

    public BigDecimal getTotalPrice() {
        return items.stream()
                // it's OrderItem's responsibility to do the multiplication!
                .map(OrderItem::getTotalPrice)
                .reduce(BigDecimal.ZERO, BigDecimal::add);
    }
}

If I did this change and my test looked like the one above, then BOOM!, my test has failed with a NullPointerException in the BigDecimal class. Pretty fragile, isn’t it?

I see at least two arguments why it’s a very, very bad thing. One is that correcting the tests every single time we change a method’s signature is annoying. The other one is that the test suite is supposed to check if I’m not breaking anything, while, instead, it stops working every single change I make!

The Myth of Complete Isolation

I believe that this style of writing mock-everything unit tests is a result of a giant misunderstanding. Unit tests, as the name suggests, are supposed to test individual software units, which can be individual classes, whole aggregates, or whatever else fits. Since we’re only interested in the proper behavior of our unit, we should isolate external dependencies such as databases, remote systems, etc. Hence, we say that unit tests are performed in isolation.

Unfortunately, when most people first brain-parse the phrase “unit tests are performed in isolation,” they understand it as “complete isolation,” i.e. we should isolate the unit from EVERYTHING. Also unfortunately, modern mocking tools are powerful enough to mock anything, including static methods and final classes. Make a combo of these two facts and you have a recipe for fragile unit tests.

This is not to say that isolation itself is bad. On the contrary, I think that the general idea of testing smaller pieces exhaustively to reduce the complexity of testing bigger ones is worthwhile. The actual problem here is coupling between the test code and the production code. The more your tests know about the inner details of your production code, the more these two are coupled together, and so the bigger the chance that changing the latter will break the former.

This brings me to the final point of this section. The notion that unit tests should be performed in complete isolation is a myth. We should use isolation to our advantage, wherever it makes our tests more manageable. At the same time, we should take coupling into account and limit the isolation in places where the pains are bigger than the gains. It’s a trade-off, as almost any programming decision.

Robust Unit Testing

Coming back to our example, we could make our Order class’s test better by giving up a little isolation and using real instances of the OrderItem class. It’s just a simple change in the way order items are created, while the rest of the test stays the same:

public class RobustOrderTest {
    // same as before:
    @Test
    public void shouldSumItemPrices() throws Exception {
        OrderItem item1 = orderItem(2, 3);
        OrderItem item2 = orderItem(4, 5);
        Order order = new Order(asList(item1, item2));
        assertEquals(BigDecimal.valueOf(26), order.getTotalPrice());
    }

    // returns real objects, not mocks! we should not care about the methods calls here!
    private OrderItem orderItem(int price, int quantity) {
        return new OrderItem(BigDecimal.valueOf(price), BigDecimal.valueOf(quantity));
    }
}

Obviously, this example is pretty naive and the code is far from what you should really write in a real project. If you start to use more real objects instead of test doubles, your tests are going to become more robust, but you will experience some new problems e.g. how to instantiate the objects for test purposes. You’re not going to copy and paste methods like orderItem all around the project, are you?

Of course, it ain’t rocket science. There are simple solutions such as static factory methods and patterns like ObjectMother or Test Data Builder to help us deal with this problem and many others that can appear. In the end, it won’t be as easy to write as bashing mocks everywhere, but it will be worthwhile.