Practical PHP Testing Patterns: Assertion Message

In the previous article of this series, we saw how assertions are the key to self-validating tests. Even if an assertion's result is only a pass/fail value, programmers and testers find very handy to associate to the fail value some kind of additional information about the failure, to allow easier debugging without insertion of var_dump() and similar statements into the code.

As I showed you in the previous article's code sample, assertions are useful the most when they fail, and in case they do, they should explain what happened so that a correction can be made quickly. An assertion that passes does not do anything; an assertion that fails shows an error.

Whata happens when tests are failing

The typical scenario is that of a large test suite, with hundreds or thousands of tests. At the time of a commit to the source code repository (or of a push if we're talking about git), the whole test suite must be green. Even if only one tests is red, the test suite is regarded as red.

Sometimes, a regression is introduced by a programming error or an unforeseen scenario, and the output of the test suite becomes something like this:

[12:30:02][giorgio@Desmond:~/txt/articles]$ phpunit assertionmessage.php 
PHPUnit 3.5.5 by Sebastian Bergmann.

FFFF

Time: 0 seconds, Memory: 3.75Mb

There were 4 failures:

1) AssertionMessagesTest::testAsserTrue
The object is not an Iterator.
Failed asserting that <boolean:false> is true.

/home/giorgio/Dropbox/txt/articles/assertionmessage.php:9

2) AssertionMessagesTest::testAssertEquals
The square root of 17 is not 4 but 4.1231056256177.
Failed asserting that <double:4.1231056256177> matches expected <integer:4>.

/home/giorgio/Dropbox/txt/articles/assertionmessage.php:17

3) AssertionMessagesTest::testAssertContainsFailsWithCustomMessage
The array does not contain 4.
Failed asserting that an array contains <integer:4>.

/home/giorgio/Dropbox/txt/articles/assertionmessage.php:24

4) AssertionMessagesTest::testAssertContains
Failed asserting that an array contains <integer:4>.

/home/giorgio/Dropbox/txt/articles/assertionmessage.php:36

FAILURES!
Tests: 4, Assertions: 4, Failures: 4.

The green tests are not noisy (which would be a smell), since they are not interesting at the moment. The red tests, which our focus is on, are shown with all the associated information. Along with the test name, the line of interest, and the kind of problem (failure/error), the assertion message is shown.

How to write a meaningful message

An assertion message can be passed as an argument of the assertion method, to be used in case of failure. Writing good assertion messages is an art, but before diving into some code to show you some messages I wrote, we can see how assertion messages are categorized in literature:

• Assertion-Identifying Message: includes information on which of the assertions in the same method caused the failure. This information can be for example the name of the checked variables. It's good to understand at a glance which assertion failed, but PHPUnit shows you the line of the test method that caused the failure, so it's not strictly necessary to include this kind of messages.
• Expectation-Describing Message: tells us what should have happened, but did not according to the data passed to the assertion method.
• Argument-Describing Message: when using assertions which are not very smart, like assertFalse() or assertTrue(), a failure would tell us by default only something like False was passed to assertTrue(). Adding an Argument-Describing Message will tell us instead The predicate on the set containing all red cars was false instead of true.

Watch the test fail

I can never say it enoug: when you write your tests, watch them fail (this mean also that each test should be written in advance with regard to the code it exercises, following the discipline of Test-Driven Development).

This practice is not only useful for avoiding false positives, which are rare anyway, but to show at least for once the assertion message. You can check that when the test fails, the error message is meaningful and the test does not cause a Fatal Error. Watch your test fail to ensure that when they'll fail due to a regression in the future, you will be pleased by how fast you'll learn what's wrong.

Example

The code sample shows how to use PHPUnit's assertion methods with the $message optional argument. Any time you use basic methods like assertTrue(), a message is mandatory. The need for $message is inversely proportional to the specificity of the assertion: for example when assertContains() fails, PHPUnit has already valuable information in the the method itself to produce a readable error message.

Note that the usage of "" (double quotes) enables fast composition of error messages via variable substitution.

<?php

class AssertionMessagesTest extends PHPUnit_Framework_TestCase
{
    public function testAsserTrue()
    {
        $object = null;
        // back in PHPUnit 3.4 where we did not have assertInstanceOf()
        $this->assertTrue($object instanceof Iterator, 'The object is not an Iterator.');
    }

    public function testAssertEquals()
    {
        $expected = 4;
        $square = 17;
        $result = sqrt($square);
        $this->assertEquals($expected, $result, "The square root of $square is not $expected but $result.");
    }

    public function testAssertContainsFailsWithCustomMessage()
    {
        $array = array(1, 2, 3);
        $testElement = 4;
        $this->assertContains($testElement, $array, "The array does not contain $testElement.");
    }

    /**
     * This Assertion Message would be enough. It will even be better in some cases,
     * since PHPUnit would display nicely $testElement even if it was not "castable" 
     * to a string (an object or array).
     */
    public function testAssertContains()
    {
        $array = array(1, 2, 3);
        $testElement = 4;
        $this->assertContains($testElement, $array);
    }
}