Applicative assertions by Mark Seemann
An exploration.
In a recent Twitter exchange, Lucas DiCioccio made an interesting observation:
"Imho the properties you want of an assertion-framework are really close (the same as?) applicative-validation: one assertion failure with multiple bullet points composed mainly from combinators."
In another branch off my initial tweet Josh McKinney pointed out the short-circuiting nature of standard assertions:
"short circuiting often causes weaker error messages in failing tests than running compound assertions. E.g.
TransferTest { a.transfer(b,50); a.shouldEqual(50); b.shouldEqual(150); // never reached? }
Most standard assertion libraries work by throwing exceptions when an assertion fails. Once you throw an exception, remaining code doesn't execute. This means that you only get the first assertion message. Further assertions are not evaluated.
Josh McKinney later gave more details about a particular scenario. Although in the general case I don't consider the short-circuiting nature of assertions to be a problem, I grant that there are cases where proper assertion composition would be useful.
Lucas DiCioccio's suggestion seems worthy of investigation.
Ongoing exploration #
I asked Lucas DiCioccio whether he'd done any work with his idea, and the day after he replied with a Haskell proof of concept.
I found the idea so interesting that I also wanted to carry out a few proofs of concept myself, perhaps within a more realistic setting.
As I'm writing this, I've reached some preliminary conclusions, but I'm also aware that they may not hold in more general cases. I'm posting what I have so far, but you should expect this exploration to evolve over time. If I find out more, I'll update this post with more articles.
- An initial proof of concept of applicative assertions in C#
- Error-accumulating composable assertions in C#
- Built-in alternatives to applicative assertions
A preliminary summary is in order. Based on the first two articles, applicative assertions look like overkill. I think, however, that it's because of the degenerate nature of the example. Some assertions are essentially one-stop verifications: Evaluate a predicate, and throw an exception if the result is false. These assertions return unit or void. Examples from xUnit include Assert.Equal, Assert.True, Assert.False, Assert.All, and Assert.DoesNotContain.
These are the kinds of assertions that the initial two articles explore.
There are other kinds of assertions that return a value in case of success. xUnit.net examples include Assert.Throws, Assert.Single, Assert.IsAssignableFrom, and some overloads of Assert.Contains. Assert.Single, for example, verifies that a collection contains only a single element. While it throws an exception if the collection is either empty or has more than one element, in the success case it returns the single value. This can be useful if you want to add more assertions based on that value.
I haven't experimented with this yet, but as far as can tell, you'll run into the following problem: If you make such an assertion return an applicative functor, you'll need some way to handle the success case. Combining it with another assertion-producing function, such as a -> Asserted e b (pseudocode) is possible with functor mapping, but will leave you with a nested functor.
You'll probably want to flatten the nested functor, which is exactly what monads do. Monads, on the other hand, short circuit, so you don't want to make your applicative assertion type a monad. Instead, you'll need to use an isomorphic monad container (Either should do) to move in and out of. Doable, but is the complexity warranted?
I realise that the above musings are abstract, and that I really should show rather than tell. I'll add some more content here if I ever collect something worthy of an article. if you ask me now, though, I consider that a bit of a toss-up.
The first two examples also suffer from being written in C#, which doesn't have good syntactic support for applicative functors. Perhaps I'll add some articles that use F# or Haskell.
Conclusion #
There's the occasional need for composable assertions. You can achieve that with an applicative functor, but the question is whether it's warranted. Could you make something simpler based on the list monad?
As I'm writing this, I don't consider that question settled. Even so, you may want to read on.
Next: An initial proof of concept of applicative assertions in C#.
Comments
I want my assertion type to be both applicative and monadic. So does Paul Loath, the creator of Language Ext, which is most clearly seen via this Validation test code. So does Alexis King (as you pointed out to me) in her Haskell Validation package, which violiates Hakell's monad type class, and which she defends here.
When I want (or typically need) short-circuiting behavior, then I use the type's monadic API. When I want "error-collecting behavior", then I use the type's applicative API.
The best syntactic support for applicative functors in C# that I have seen is in Langauge Ext. A comment explains in that same Validation test how it works, and the line after the comment shows it in action.
Tyson, thank you for writing. Whether or not you want to enable monadic short-circuiting for assertions or validations depends, I think, on 'developer ergonomics'. It's a trade-off mainly between ease and simplicity as outlined by Rich Hickey. Enabling a monadic API for something that isn't naturally monadic does indeed provide ease of use, in that the compositional capabilities of a monad are readily 'at hand'.
If you don't have that capability you'll have to map back and forth between, say,
ValidationandEither(if using the validation package). This is tedious, but explicit.Making validation or assertions monadic makes it easier to compose nested values, but also (in my experience) makes it easier to make mistakes, in the sense that you (or a colleague) may think that the behaviour is error-collecting, whereas in reality it's short-circuiting.
In the end, the trade-off may reduce to how much you trust yourself (and co-workers) to steer clear of mistakes, and how important it is to avoid errors. In this case, how important is it to collect the errors, rather than short-circuiting?
You can choose one alternative or the other by weighing such concerns.