Practical PHP Testing Patterns: Unfinished Test Assertion

Sometimes we are in the middle of writing a test and another possible, unrelated test comes to mind. Now we are in a tough situation: Test-Driven Development tells us to go one step at a time for good reasons. But if we just wait before inserting that test, we can forget it even if it is a nasty corner case.

We need an approach for storing these tests somewhere, hoping that in the future they could be implemented (or discarded upon deeper analysis).

What we want

We don't want to write the test on a Post-it note, as it will be probably forgotten just like in our own brain RAM. A nice thing would be to keep this new test in source control, even if it will fail for now.

If we are going to introduce it in a Testcase Class, we don't want side effects on already existing tests (no explosion of the suite or compilation errors).
A first approximation of the solution can be an empty Test Method:

public function testThatXDoesY() {}

or a slightly more sophisticated:

public function testThatXDoesY()
{
    //call method Z and check W is equal to K
}

However, we want to be told that this test is not green. An empty test is a false positive: we see a dot in the output, and we could forget to go over that test again to actually write some code in it.

Thus another approximation may be just throwing an exception to make the test fail:

public function testThatXDoesY()
{
    throw new Exception('Write me!');
}

But often we want to distinguish between tests that we still have to write and regressions. In that case, we want a clear message that tells us that this test is still incomplete (not failing: just incomplete) and what we can do to fix it.

In all these scenarios, we can introduce an Unfinished Test Assertion, which is an assertion that always fails and provide a meaningful message for the failure. Some of these assertions fail in special ways (see the Implementation in PHPUnit section.)

It's also common sense to include Unfinished Test Assertions when tests are generated (una tantum) by some other process, and implemented one at the time by developers. Even when these tests are spreaded in your suite, you can always track them by running it as a whole.

Implementation in PHPUnit

Basically, inside a PHPUnit_Framework_TestCase you have three methods to execute an Unfinished Test Assertion:

• $this->fail('message') displays a F instead of a . in the results, and implies a red bar. It's just a semantic way to execute $this->assertTrue(false).
• $this->markTestSkipped('message') displays an S in the results, and implies a yellow bar instead of a green one.
• $this->markTestIncomplete('message') displays an I in the results, and also implies a yellow bar.

Of course you can wrap these basic methods with yours.

The difference between skipped and incomplete tests is only semantic, not functional: incomplete tests lack code, skipped are complete but cannot be run in the current environment or due to some other test already failing.

The difference between $this->fail() and the other methods is instead clear: even one failed test results in a red bar, while skipped and incomplete tests make the bar yellow, which is a less serious issue.

Example

I have included a sample Testcase Class which shows usage of the various predefined methods, plus a custom one.

If you want to actually see the messages of skipped and incomplete tests, you must run phpunit --verbose instead of simply phpunit.

The code is also available on the Github repository for this series: https://github.com/giorgiosironi/practical-php-testing-patterns.

<?php
class UnfinishedTestAssertions extends PHPUnit_Framework_TestCase
{
    public function testThatFails()
    {
        $this->fail('Write this test, you should check that X does Y.');
    }

    public function testThatIsSkipped()
    {
        $this->markTestSkipped('To execute this test, you need an active Internet connection.');
    }

    public function testThatIsIncomplete()
    {
        $fixture = new ArrayObject();
        $fixture['key'] = 'value';

        $this->markTestIncomplete('This test needs to be finished: the act and assert parts are missing.');
    }

    public function testThatIsDeclaredIncompleteWithCustomUnifinishedTestAssertion()
    {
        $this->assertionWillBeNeeded();
    }

    private function assertionWillBeNeeded()
    {
        $this->markTestIncomplete('The test lacks an assert phase. Write it.');
    }
}