Composing pure Haskell assertions by Mark Seemann
With HUnit and QuickCheck examples.
A question had been in the back of my mind for a long time, but I always got caught up in something seemingly more important, so I didn't get around to investigate until recently. It's simply this:
How do you compose pure assertions in HUnit or QuickCheck?
Let me explain what I mean, and why this isn't quite as straightforward as it may sound.
Assertions as statements #
What do I mean by composing assertions? Really nothing more than wanting to verify more than a single outcome of a test.
If you're used to writing test assertions in imperative languages like C#, Java, Python, or JavaScript, you think nothing of it. Just write an assertion on one line, and the next assertion on the next line.
If you're writing impure Haskell, you can also do that.
"CLRS example" ~: do p :: IOArray Int Int <- newListArray (1, 10) [1, 5, 8, 9, 10, 17, 17, 20, 24, 30] (r, s) <- cutRod p 10 actualRevenue <- getElems r actualSizes <- getElems s let expectedRevenue = [0, 1, 5, 8, 10, 13, 17, 18, 22, 25, 30] let expectedSizes = [1, 2, 3, 2, 2, 6, 1, 2, 3, 10] expectedRevenue @=? actualRevenue expectedSizes @=? actualSizes
This example is an inlined HUnit test which tests the impure cutRod variation. The two final statements are assertions that use the @=? assertion operator. The value on the left side is the expected value, and to the right goes the actual value. This operator returns a type called Assertion, which turns out to be nothing but an alias for IO ().
In other words, those assertions are impure actions, and they work similarly to assertions in imperative languages. If the actual value passes the assertion, nothing happens and execution moves on to the assertion on the next line. If, on the other hand, the assertion fails, execution short-circuits, and an error is reported.
Imperative languages typically throw exceptions to achieve that behaviour. Even Unquote does this. Exactly how HUnit does it I don't know; I haven't looked under the hood.
You can do the same with Test.QuickCheck.Monadic:
testProperty "cutRod returns correct arrays" $ \ p -> monadicIO $ do let n = length p p' :: IOArray Int Int <- run $ newListArray (1, n) p (r, s) :: (IOArray Int Int, IOArray Int Int) <- run $ cutRod p' n actualRevenue <- run $ getElems r actualSizes <- run $ getElems s assertWith (length actualRevenue == n + 1) "Revenue length is incorrect" assertWith (length actualSizes == n) "Size length is incorrect" assertWith (all (\i -> 0 <= i && i <= n) actualSizes) "Sizes are not all in [1..n]"
Like the previous example, you can repeatedly call assertWith, since this action, too, is a statement that returns no value.
So far, so good.
Composing assertions #
What if, however, you want to write tests as pure functions?
Pure functions are composed from expressions, while statements aren't allowed (or, at least, ineffective, and subject to be optimized away by a compiler). In other words, the above strategy isn't going to work. If you want to write more than one assertion, you need to figure out how they compose.
The naive answer might be to use logical conjunction (AKA Boolean and). Write one assertion as a Boolean expression, another assertion as another Boolean expression, and just compose them using the standard 'and' operator. In Haskell, that would be &&.
This works to a fashion, but has a major drawback. If such a composed assertion fails, it doesn't tell you why. All you know is that the entire Boolean expression evaluated to False.
This is the reason that most testing libraries come with explicit assertion APIs. In HUnit, you may wish to use the ~=? operator, and in QuickCheck the === operator.
The question, however, is how they compose. Ideally, assertions should compose applicatively, but I've never seen that in the wild. If not, look for a monoid, or at least a semigroup.
Let's do that for both HUnit and QuickCheck.
Composing HUnit assertions #
My favourite HUnit assertion is the ~=? operator, which has the (simplified) type a > a -> Test. In other words, an expression like expectedRevenue ~=? actualRevenue has the type Test. The question, then, is: How does Test compose?
Not that well, I'm afraid, but I find the following workable. You can compose one or more Test values with the TestList constructor, but if you're already using the ~: operator, as I usually do (see below), then you just need a Testable instance, and it turns out that a list of Testable values is itself a Testable instance. This means that you can write a pure unit test and compose ~=? like this:
"CLRS example" ~: let p = [0, 1, 5, 8, 9, 10, 17, 17, 20, 24, 30] :: [Int] (r, s) = cutRod p 10 actualRevenue = Map.elems r actualSizes = Map.elems s expectedRevenue = [0, 1, 5, 8, 10, 13, 17, 18, 22, 25, 30] expectedSizes = [1, 2, 3, 2, 2, 6, 1, 2, 3, 10] in [expectedRevenue ~=? actualRevenue, expectedSizes ~=? actualSizes]
This is a refactoring of the above test, now as a pure function, because it tests the pure variation of cutRod. Notice that the two assertions are simply returned as a list.
While this has enough syntactical elegance to satisfy me, it does have the disadvantage that it actually creates two test cases. One that runs with the first assertion, and one that executes with the second:
:CLRS example: : [OK] : [OK]
In most cases this is unlikely to be a problem, but it could be if the test performs a resource-intensive computation. Each assertion you add makes it run one more time.
A test like the one shown here is so 'small' that this is rarely much of an issue. On the other hand, a property-based testing library might stress a System Under Test more, so fortunately, QuickCheck assertions compose better than HUnit assertions.
Composing QuickCheck assertions #
The === operator has the (simplified) type a -> a -> Property. Hoogling for a combinator with the type Property -> Property -> Property doesn't reveal anything useful, but fortunately it turns out that for running QuickCheck properties, all you really need is a Testable instance (not the same Testable as HUnit defines). And lo and behold! The .&&. operator is just what we need. That, or the conjoin function, if you have more than two assertions to combine, as in this example:
testProperty "cutRod returns correct arrays" $ \ p -> do let n = length p let p' = 0 : p -- Ensure the first element is 0 let (r, s) :: (Map Int Int, Map Int Int) = cutRod p' n let actualRevenue = Map.elems r let actualSizes = Map.elems s conjoin [ length actualRevenue === n + 1, length actualSizes === n, counterexample "Sizes are not all in [0..n]" $ all (\i -> 0 <= i && i <= n) actualSizes ]
The .&&. operator is actually a bit more flexible than conjoin, but due to operator precedence and indentation rules, trying to chain those three assertions with .&&. is less elegant than using conjoin. In this case.
Conclusion #
In imperative languages, composing test assertions is as simple as writing one assertion after another. Since assertions are statements, and imperative languages allow you to sequence statements, this is such a trivial way to compose assertions that you've probably never given it much thought.
Pure programs, however, are not composed from statements, but rather from expressions. A pure assertion is an expression that returns a value, so if you want to compose two or more pure assertions, you need to figure out how to compose the values that the assertions return.
Ideally, assertions should compose as applicative functors, but they rarely do. Instead, you'll have to go looking for combinators that enable you to combine two or more of a test library's built-in assertions. In this article, you've seen how to compose assertions in HUnit and QuickCheck.
