Built-in alternatives to applicative assertions by Mark Seemann
Why make things so complicated?
Several readers reacted to my small article series on applicative assertions, pointing out that error-collecting assertions are already supported in more than one unit-testing framework.
"In the Java world this seems similar to the result gained by Soft Assertions in AssertJ. https://assertj.github.io/doc/#assertj-c... if you’re after a target for functionality (without the adventures through monad land)"
While I'm not familiar with the details of Java unit-testing frameworks, the situation is similar in .NET, it turns out.
"Did you know there is Assert.Multiple in NUnit and now also in xUnit .Net? It seems to have quite an overlap with what you're doing here.
"For a quick overview, I found this blogpost helpful: https://www.thomasbogholm.net/2021/11/25/xunit-2-4-2-pre-multiple-asserts-in-one-test/"
I'm not surprised to learn that something like this exists, but let's take a quick look.
NUnit Assert.Multiple #
Let's begin with NUnit, as this seems to be the first .NET unit-testing framework to support error-collecting assertions. As a beginning, the documentation example works as it's supposed to:
[Test] public void ComplexNumberTest() { ComplexNumber result = SomeCalculation(); Assert.Multiple(() => { Assert.AreEqual(5.2, result.RealPart, "Real part"); Assert.AreEqual(3.9, result.ImaginaryPart, "Imaginary part"); }); }
When you run the test, it fails (as expected) with this error message:
Message: Multiple failures or warnings in test: 1) Real part Expected: 5.2000000000000002d But was: 5.0999999999999996d 2) Imaginary part Expected: 3.8999999999999999d But was: 4.0d
That seems to work well enough, but how does it actually work? I'm not interested in reading the NUnit source code - after all, the concept of encapsulation is that one should be able to make use of the capabilities of an object without knowing all implementation details. Instead, I'll guess: Perhaps Assert.Multiple executes the code block in a try/catch block and collects the various exceptions thrown by the nested assertions.
Does it catch all exception types, or only a subset?
Let's try with the kind of composed assertion that I previously investigated:
[Test] public void HttpExample() { var deleteResp = new HttpResponseMessage(HttpStatusCode.BadRequest); var getResp = new HttpResponseMessage(HttpStatusCode.OK); Assert.Multiple(() => { deleteResp.EnsureSuccessStatusCode(); Assert.That(getResp.StatusCode, Is.EqualTo(HttpStatusCode.NotFound)); }); }
This test fails (again, as expected). What's the error message?
Message: System.Net.Http.HttpRequestException :↩ Response status code does not indicate success: 400 (Bad Request).
(I've wrapped the result over multiple lines for readability. The ↩ symbol indicates where I've wrapped the text. I'll do that again later in this article.)
Notice that I'm using EnsureSuccessStatusCode as an assertion. This seems to spoil the behaviour of Assert.Multiple. It only reports the first status code error, but not the second one.
I admit that I don't fully understand what's going on here. In fact, I have taken a cursory glance at the relevant NUnit source code without being enlightened.
One hypothesis might be that NUnit assertions throw special Exception sub-types that Assert.Multiple catch. In order to test that, I wrote a few more tests in F# with Unquote, assuming that, since Unquote hardly throws NUnit exceptions, the behaviour might be similar to above.
[<Test>] let Test4 () = let x = 1 let y = 2 let z = 3 Assert.Multiple (fun () -> x =! y y =! z)
The =! operator is an Unquote operator that I usually read as must equal. How does that error message look?
Message: Multiple failures or warnings in test: 1) 1 = 2 false 2) 2 = 3 false
Somehow, Assert.Multiple understands Unquote error messages, but not HttpRequestException. As I wrote, I don't fully understand why it behaves this way. To a degree, I'm intellectually curious enough that I'd like to know. On the other hand, from a maintainability perspective, as a user of NUnit, I shouldn't have to understand such details.
xUnit.net Assert.Multiple #
How fares the xUnit.net port of Assert.Multiple?
[Fact] public void HttpExample() { var deleteResp = new HttpResponseMessage(HttpStatusCode.BadRequest); var getResp = new HttpResponseMessage(HttpStatusCode.OK); Assert.Multiple( () => deleteResp.EnsureSuccessStatusCode(), () => Assert.Equal(HttpStatusCode.NotFound, getResp.StatusCode)); }
The API is, you'll notice, not quite identical. Where the NUnit Assert.Multiple method takes a single delegate as input, the xUnit.net method takes an array of actions. The difference is not only at the level of API; the behaviour is different, too:
Message: Multiple failures were encountered: ---- System.Net.Http.HttpRequestException :↩ Response status code does not indicate success: 400 (Bad Request). ---- Assert.Equal() Failure Expected: NotFound Actual: OK
This error message reports both problems, as we'd like it to do.
I also tried writing equivalent tests in F#, with and without Unquote, and they behave consistently with this result.
If I had to use something like Assert.Multiple, I'd trust the xUnit.net variant more than NUnit's implementation.
Assertion scopes #
Apparently, Fluent Assertions offers yet another alternative.
"Hey @ploeh, been reading your applicative assertion series. I recently discovered Assertion Scopes, so I'm wondering what is your take on them since it seems to me they are solving this problem in C# already. https://fluentassertions.com/introduction#assertion-scopes"
The linked documentation contains this example:
[Fact] public void DocExample() { using (new AssertionScope()) { 5.Should().Be(10); "Actual".Should().Be("Expected"); } }
It fails in the expected manner:
Message: Expected value to be 10, but found 5 (difference of -5). Expected string to be "Expected" with a length of 8, but "Actual" has a length of 6,↩ differs near "Act" (index 0).
How does it fare when subjected to the EnsureSuccessStatusCode test?
[Fact] public void HttpExample() { var deleteResp = new HttpResponseMessage(HttpStatusCode.BadRequest); var getResp = new HttpResponseMessage(HttpStatusCode.OK); using (new AssertionScope()) { deleteResp.EnsureSuccessStatusCode(); getResp.StatusCode.Should().Be(HttpStatusCode.NotFound); } }
That test produces this error output:
Message: System.Net.Http.HttpRequestException :↩ Response status code does not indicate success: 400 (Bad Request).
Again, EnsureSuccessStatusCode prevents further assertions from being evaluated. I can't say that I'm that surprised.
Implicit or explicit #
You might protest that using EnsureSuccessStatusCode and treating the resulting HttpRequestException as an assertion is unfair and unrealistic. Possibly. As usual, such considerations are subject to a multitude of considerations, and there's no one-size-fits-all answer.
My intent with this article isn't to attack or belittle the APIs I've examined. Rather, I wanted to explore their boundaries by stress-testing them. That's one way to gain a better understanding. Being aware of an API's limitations and quirks can prevent subtle bugs.
Even if you'd never use EnsureSuccessStatusCode as an assertion, perhaps you or a colleague might inadvertently do something to the same effect.
I'm not surprised that both NUnit's Assert.Multiple and Fluent Assertions' AssertionScope behaves in a less consistent manner than xUnit.net's Assert.Multiple. The clue is in the API.
The xUnit.net API looks like this:
public static void Multiple(params Action[] checks)
Notice that each assertion is explicitly a separate action. This enables the implementation to isolate it and treat it independently of other actions.
Neither the NUnit nor the Fluent Assertions API is that explicit. Instead, you can write arbitrary code inside the 'scope' of multiple assertions. For AssertionScope, the notion of a 'scope' is plain to see. For the NUnit API it's more implicit, but the scope is effectively the extent of the method:
public static void Multiple(TestDelegate testDelegate)
That testDelegate can have as many (nested, even) assertions as you'd like, so the Multiple implementation needs to somehow demarcate when it begins and when it ends.
The testDelegate can be implemented in a different file, or even in a different library, and it has no way to communicate or coordinate with its surrounding scope. This reminds me of an Ambient Context, an idiom that Steven van Deursen convinced me was an anti-pattern. The surrounding context changes the behaviour of the code block it surrounds, and it's quite implicit.
Explicit is better than implicit.
The xUnit.net API, at least, looks a bit saner. Still, this kind of API is quirky enough that it reminds me of Greenspun's tenth rule; that these APIs are ad-hoc, informally-specified, bug-ridden, slow implementations of half of applicative functors.
Conclusion #
Not surprisingly, popular unit-testing and assertion libraries come with facilities to compose assertions. Also, not surprisingly, these APIs are crude and require you to learn their implementation details.
Would I use them if I had to? I probably would. As Rich Hickey put it, they're already at hand. That makes them easy, but not necessarily simple. APIs that compel you to learn their internal implementation details aren't simple.
Universal abstractions, on the other hand, you only have to learn one time. Once you understand what an applicative functor is, you know what to expect from it, and which capabilities it has.
In languages with good support for applicative functors, I would favour an assertion API based on that abstraction, if given a choice. At the moment, though, that's not much of an option. Even HUnit assertions are based on side effects.
Comments
Just a reminder: in .NET, method's execution cannot be resumed after an exception is thrown, there is just simply no way to do this, at all. Which means that NUnit's Assert.Multiple absolutely cannot work the way you guess it probably does, by running the delegate and resuming its execution after it throws an exception until the delegate returns.
How could it work then? Well, considering that documentation to almost every Assert's method has "Returns without throwing an exception when inside a multiple assert block" line in it, I would assume that Assert.Multiple sets a global flag which makes actual assertions to store the failures in some global hidden context instead on throwing them, then runs the delegate and after it finishes or throws, collects and clears all those failures from the context and resets the global flag.
Cursory inspection of NUnit's source code supports this idea, except that apparently it's not just a boolean flag but a "depth" counter; and assertions report the failures just the way I've speculated. I personally hate such side-channels but you have to admit, they allow for some nifty, seemingly impossible magical tricks (a.k.a. "spooky action at the distance").
Also, why do you assume that Unquote would not throw NUnit's assertions? It literally has "Unquote integrates configuration-free with all exception-based unit testing frameworks including xUnit.net, NUnit, MbUnit, Fuchu, and MSTest" in its README, and indeed, if you look at its source code, you'll see that at runtime it tries to locate any testing framework it's aware of and use its assertions. More funny party tricks, this time with reflection!
I understand that after working in more pure/functional programming environments one does start to slowly forget about those terrible things, but: those horrorterrors still exist, and people keep making more of them. Now, if you can, have a good night :)
Joker_vD, thank you for explaining those details. I admit that I hadn't thought too deeply about implementation details, for the reasons I briefly mentioned in the post.
Yes, that summarises my current thinking well, I'm afraid.
NUnit has Assert.DoesNotThrow and Fluent Assertions has .Should().NotThrow(). I did not check Fluent Assertions, but NUnit does gather failures of Assert.DoesNotThrow inside Assert.Multiple into a multi-error report. One might argue that asserting that a delegate should not throw is another application of the "explicit is better than implicit" philosophy. Here's what Fluent Assertions has to say on that matter:
As a side note, you might also want to take a look on NUnits Assert.That syntax. It allows to construct complex conditions tested against a single actual value:
A failure is then reported like this:
Max, thank you for writing. I have to admit that I never understood the point of NUnit's constraint model, but your example clearly illustrates how it may be useful. It enables you to compose assertions.
It's interesting to try to understand the underlying reason for that. I took a cursory glance at that
IResolveConstraintAPI, and as far as I can tell, it may form a monoid (I'm not entirely sure about theConstraintStatusenum, but even so, it may be 'close enough' to be composable).I can see how that may be useful when making assertions against complex objects (i.e. object composed from other objects).
In xUnit.net you'd typically address that problem with custom IEqualityComparers. This is more verbose, but also strikes me as more reusable. One disadvantage of that approach, however, is that when tests fail, the assertion message is typically useless.
This is the reason I favour Unquote: Instead of inventing a Boolean algebra(?) from scratch, it uses the existing language and still gives you good error messages. Alas, that only works in F#.
In general, though, I'm inclined to think that all of these APIs address symptoms rather than solve real problems. Granted, they're useful whenever you need to make assertions against values that you don't control, but for your own APIs, a simpler solution is to model values as immutable data with structural equality.
Another question is whether aiming for clear assertion messages is optimising for the right concern. At least with TDD, I don't think that it is.