Kotlintest and Property-Based Testing

I was very happy to see that Kotlintest, a port of the excellent scalatest in Kotlin, supports property-based testing.

I was introduced to property-based testing through the excellent "Functional programming in Scala" book.

The idea behind it is simple — the behavior of a program is described as a property, and the testing framework generates random data to validate the property. This is best illustrated with an example using the excellent scalacheck library:

import org.scalacheck.Prop.forAll
import org.scalacheck.Properties

object ListSpecification extends Properties("List") {
    property("reversing a list twice should return the list") = forAll { (a: List[Int]) =>
        a.reverse.reverse == a
    }
}

scalacheck would generate a random list (of integers) of varying sizes and would validate that this property holds for the lists. A similar specification expressed through Kotlintest looks like this:

import io.kotlintest.properties.forAll
import io.kotlintest.specs.StringSpec

class ListSpecification : StringSpec({
    "reversing a list twice should return the list" {
        forAll{ list: List<Int> ->
            list.reversed().reversed().toList() == list
        }
    }
})

If the generators have to be a little more constrained, say if we wanted to test this behavior on lists of integers in the range 1-1000, then an explicit generator can be passed in the following way, again starting with scalacheck:

import org.scalacheck.Prop.forAll
import org.scalacheck.{Gen, Properties}

object ListSpecification extends Properties("List") {
    val intList = Gen.listOf(Gen.choose(1, 1000))
    property("reversing a list twice should return the list") = forAll(intList) { (a: List[Int]) =>
        a.reverse.reverse == a
    }
}

And the equivalent kotlintest code:

import io.kotlintest.properties.Gen
import io.kotlintest.properties.forAll
import io.kotlintest.specs.StringSpec

class BehaviorOfListSpecs : StringSpec({
    "reversing a list twice should return the list" {
        val intList = Gen.list(Gen.choose(1, 1000))

        forAll(intList) { list ->
            list.reversed().reversed().toList() == list
        }
    }
})

Given this, let me now jump onto another example from the scalacheck site, this time to illustrate a failure:

import org.scalacheck.Prop.forAll
import org.scalacheck.Properties

object StringSpecification extends Properties("String") {

    property("startsWith") = forAll { (a: String, b: String) =>
        (a + b).startsWith(a)
    }

    property("concatenate") = forAll { (a: String, b: String) =>
        (a + b).length > a.length && (a + b).length > b.length
    }

    property("substring") = forAll { (a: String, b: String, c: String) =>
        (a + b + c).substring(a.length, a.length + b.length) == b
    }
}

The second property described above is wrong — if two strings are concatenated together, they are ALWAYS larger than each of the parts. This is not true if one of the strings is blank.

If I were to run this test using scalacheck, it correctly catches this wrongly specified behavior:

+ String.startsWith: OK, passed 100 tests.
! String.concatenate: Falsified after 0 passed tests.
> ARG_0: ""
> ARG_1: ""
+ String.substring: OK, passed 100 tests.
Found 1 failing properties.

An equivalent in Kotlintest is the following:

import io.kotlintest.properties.forAll
import io.kotlintest.specs.StringSpec

class StringSpecification : StringSpec({
    "startsWith" {
        forAll { a: String, b: String ->
            (a + b).startsWith(a)
        }
    }

    "concatenate" {
        forAll { a: String, b: String ->
            (a + b).length > a.length && (a + b).length > b.length
        }
    }

    "substring" {
        forAll { a: String, b: String, c: String ->
            (a + b + c).substring(a.length, a.length + b.length) == b
        }
    }
})

On running, it correctly catches the issue with concatenation and produces the following result:

java.lang.AssertionError: Property failed for

Y{_DZ<vGnzLQHf9|3$i|UE,;!%8^SRF;JX%EH+<5d:p`Y7dxAd;I+J5LB/:O)

 at io.kotlintest.properties.PropertyTestingKt.forAll(PropertyTesting.kt:27)

However, there is an issue here — scalacheck found a simpler failure case. It does this with a process called "Test Case minimization," where in the case of a failure, it tries to find the smallest test case that can fail, something that the Kotlintest can learn from.

There are other features where Kotlintest lags in respect to scalacheck, a big one being able to combine generators:

case class Person(name: String, age: Int)

val genPerson = for {
    name <- Gen.alphaStr
    age <- Gen.choose(1, 50)
} yield Person(name, age)

genPerson.sample

However, all in all, I have found the DSL of Kotlintest and its support for property-based testing to be a good start so far, and I look forward to seeing how this library will evolve over time.

If you want to play with these samples a little more, it is available in my GitHub repo here.