Ad hoc Arbitraries with FsCheck.Xunit by Mark Seemann

When using FsCheck with xUnit.net, you can define ad hoc Arbitraries in-line in your test functions.

Writing properties with FsCheck and using xUnit.net as a test host is a nice combination. Properties are written as normal functions annotated with the Property attribute:

[<Property>]
let ``findNeighbors returns 8 cells`` (cell : int * int) =
    let actual : Set<int * int> = findNeighbors cell
    8 =! actual.Count

FsCheck takes care of generating values for the cell argument. This works well in many cases, but sometimes you need a bit more control over the range of values being generated by FsCheck.

Motivating example #

You may already have guessed it from the above code snippet, but the example for this article is a property-based approach to Conway's Game of Life. One of the properties of the game is that any live cell with more than three live neighbours dies.

This means that you have to write a property where one of the values is the number of live neighbours. That number must be a (randomly generated) number between 4 and 8 (including both ends), because the maximum number of neighbours in a cell grid is eight. How do you get FsCheck to give you a number between 4 and 8?

Crude solution: conditional property #

There are various solutions to the problem of getting fine control over certain values generated by FsCheck, but the ones I first learned turn out to be problematic.

One attempt is to use conditional properties, where you supply a boolean expression, and FsCheck will throw away all properties that fail the boolean test:

[<Property>]
let ``Any live cell with more than three live neighbors dies``
    (cell : int * int)
    (neighborCount : int) =
    (3 < neighborCount && neighborCount <= 8) ==> lazy
 
    let neighborCells = findNeighbors cell |> pickRandom neighborCount
    let actual = calculateNextState (cell :: neighborCells) cell
    Dead =! actual

The ==> operator is an FsCheck-specific operator that indicates that FsCheck should disregard all properties where the input arguments don't satisfy the boolean condition on the left side. The use of lazy ensures that FsCheck will only attempt to evaluate the property when the condition is true.

That's simple and tidy, but unfortunately doesn't work. If you attempt to run this property, you'll get a result like this:

Test 'Ploeh.Katas.GameOfLifeProperties.Any live cell with more than three live neighbors dies' failed: 
Arguments exhausted after 34 tests.

The reason is that FsCheck generates random integers for neighborCount, and most of these values (statistically) fall outside of the range 4-8. After a (default) maximum of 1000 attempts, FsCheck gives up, because at that time, it's only managed to find 34 values that satisfy the condition, but it wants to find 100.

What a disappointment.

Crude solution: custom Arbitrary #

Another apparent solution is to define a custom Arbitrary for FsCheck.Xunit. The mechanism is to define a static class with your custom rule, and register that with the Property attribute. The class must have a static method that returns an Arbitrary<'a>.

In this particular example, you'll need to define a custom Arbitrary that only picks random numbers between 4 and 8. That's easy, but there's a catch: if you change the way int values are generated, you're also going to impact the generated cell values, because a cell here is an int * int tuple.

Since you only need a small number, you can cheat and customize byte values instead:

type ByteBetween1and8 =
    static member Byte () = Gen.elements [1uy .. 8uy] |> Arb.fromGen
 
[<Property(Arbitrary = [| typeof<ByteBetween1and8> |])>]
let ``Any live cell with more than three live neighbors dies``
    (cell : int * int)
    (neighborCount : byte) =
    neighborCount > 3uy ==> lazy
 
    let neighborCells = findNeighbors cell |> pickRandom (int neighborCount)
    let actual = calculateNextState (cell :: neighborCells) cell
    Dead =! actual

The ByteBetween1and8 type is a static class with a single static member that returns Arbitrary<byte>. By using the Arbitrary property of the Property attribute (!), you can register this custom Arbitrary with the Property.

This 'solution' side-steps the issue by using a substitute data type instead of the desired data type. There are several problems with this:

• You have to convert the byte value back to an integer in order to use it with the System Under Test: int neighborCount.
• 3uy is less readable than 3.
• A newcomer will wonder why neighborCount is a byte instead of an int.
• You can't generalise this solution. It works because F# has more than one number type, but if you need to generate strings, you're out of luck: there's only one (normal) type of string in .NET.

Next to these issues with using bytes instead of ints, there's the larger problem that it's awkward to have to define an entire new type (ByteBetween1and8) for this purpose. Finally, this mechanism isn't type-safe. ByteBetween1and8 doesn't implement any interface. It doesn't have to, because the Arbitrary property on the Property attribute is just an array of Type instances. They can be any type, and your code will compile, but if one of those Types don't meet the requirements, an exception will be thrown at run-time.

This way works, but is hardly elegant or safe.

Ad hoc, in-line Arbitraries #

There's a safe way to define ad hoc, in-line Arbitraries with FsCheck.Xunit:

[<Property>]
let ``Any live cell with > 3 live neighbors dies`` (cell : int * int) =
    let nc = Gen.elements [4..8] |> Arb.fromGen
    Prop.forAll nc (fun neighborCount ->
        let liveNeighbors =
            cell
            |> findNeighbors
            |> shuffle
            |> Seq.take neighborCount
            |> Seq.toList
        
        let actual : State =
            calculateNextState (cell :: liveNeighbors |> shuffle |> set) cell
 
        Dead =! actual)

Using Prop.forAll enables you to execute your property with a custom Arbitrary, but mixed with the 'normally' generated values that the function receives via its arguments. In this example, nc is an Arbitrary<int>. Notice how it's explicitly used to populate the neighborCount value of the property, whereas the cell value arrives via normal means.

This is type-safe, because nc is an Arbitrary<int>, which means that neighborCount is statically inferred to be an int.

If you need more than a single ad hoc Arbitrary, you can always create Arbitraries for each of them, and then use Gen.map2, Gen.map3, and so on, to turn those individual Arbitraries into a single Arbitrary of a tuple. You can then use that tuple with Prop.forAll.

Summary #

FsCheck is a well-designed library that you can combine in lots of interesting ways. In this article you learned how to use Prop.forAll to evaluate a property with a mix of normal, arbitrarily generated values, and an ad hoc, in-line Arbitrary.

Addendum 2016-03-01: You can write such properties slightly better using the backward pipe operator.

• ← Previous
• Archive
• Next →