Don't Sleep When You Test!

When I started working as a Java Developer, my teammate and I got a first task of repairing all broken tests (great task for new starters!) in some old project. Replacing some old configurations and upgrading a few libraries helped make the test's status green, but there was another problem. End-to-end test suites took a few hours to pass, making it impossible to get the feedback for a new change quickly. Sometimes builds ran for a few hours and hung, so E2E tests became excluded from the feedback loop at the end. 

Most of the tests from that suite did the same thing: 

• Invoke a few commands on the mainframe through HTTP interface
• Sleep for 30 seconds (probably just to be extra safe) after each command and check if the result was persisted to database by another system. 

It turned out, that first try to reduce the sleep time to 15 seconds by my teammate, didn’t make any tests fail, so that was a nice first try. But what was the optimal wait time for each test?

After some time, we tried another approach. If the flow has a asynchronous nature then why not test it asynchronously?

I introduced this simple class (or at least class which does the same): 

public class Await {

    private static final BiPredicate<Long, Long> notTimedOut =
            (start, timeoutMillis) -> System.currentTimeMillis() - start < timeoutMillis;

    public static void await(Supplier<AssertionResult> assertionVerifier, long timeoutMillis) {
        final long startTime = System.currentTimeMillis();

        Stream.generate(assertionVerifier)
                .takeWhile(result -> notTimedOut.test(startTime, timeoutMillis) || fail(result))
                .filter(AssertionResult::isOK)
                .findFirst();
    }

    public static boolean fail(AssertionResult result) {
        throw new RuntimeException("Assertion not passed after timeout, problem: " + result.optionalMessage);
    }
}

Method await checks some assertion in a loop until either it won’t throw AssertionError or the timeout is passed. This assertion is usually some state we are waiting for that should be ready in the nearest future. If it is not ready after the timeout, the last message from the AssertionError  will be thrown wrapped as a RuntimeException. This greatly reduces test time because if the result is ready earlier, we don’t have to spend unnecessary time in the sleep method.

To represent the assertion result we can use classes (minimal implementation):

enum Status { OK, ERROR }

class AssertionResult {

    public static final AssertionResult OK = new AssertionResult(Status.OK, "");

    public final Status status;

    public final String optionalMessage;

    private AssertionResult(Status status, String optionalMessage) {
        this.status = status;
        this.optionalMessage = optionalMessage;
    }

    public static AssertionResult error(String optionalMessage) {
        return new AssertionResult(Status.ERROR, optionalMessage);
    }

    public boolean isOK() {
        return status == Status.OK;
    }
}

We also need to prepare our AssertionResult suppliers - wrappers which fetch the data and invoke assertions on the result. Usually it is possible to define generic supplier if the way we fetch the data is similar. For example, we can write this builder method:

public static Supplier<AssertionResult> dbDataAssertion(Consumer<String> userAssertion, long userId) {
    return () -> {
        String usernameFromDB = db.getUserName(userId); //or any other more generic version
        try {
            userAssertion.accept(usernameFromDB);
            return AssertionResult.OK;
        } catch (AssertionError assertionError) {
            return AssertionResult.error(assertionError.getMessage());
        }
    };
}

We are ready to replace  Thread.sleep()  with call to Await in our tests:

...
long timeoutMillis = 1000L;
long userId = 1234L;
...
//then
Await.await(
        dbDataAssertion(userName -> assertEquals(4, userName.length()), userId),
        timeoutMillis); 

Conclusion

Don’t overuse  Thread.sleep()  in your test code. According to this Google research it is one of the main reasons of flaky tests. Asynchronous approach may complicate your tests, but the time savings can be tremendous.  If you are looking for some existing implementations there is a nice class in Spock which does the same.