EnsureSuccessStatusCode as an assertion by Mark Seemann
It finally dawned on me that EnsureSuccessStatusCode is a fine test assertion.
Okay, I admit that sometimes I can be too literal and rigid in my thinking. At least as far back as 2012 I've been doing outside-in TDD against self-hosted HTTP APIs. When you do that, you'll regularly need to verify that an HTTP response is successful.
Assert equality #
You can, of course, write an assertion like this:
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
If the assertion fails, it'll produce a comprehensible message like this:
Assert.Equal() Failure Expected: OK Actual: BadRequest
What's not to like?
Assert success #
The problem with testing strict equality in this context is that it's often too narrow a definition of success.
You may start by returning 200 OK as response to a POST request, but then later evolve the API to return 201 Created or 202 Accepted. Those status codes still indicate success, and are typically accompanied by more information (e.g. a Location header). Thus, evolving a REST API from returning 200 OK to 201 Created or 202 Accepted isn't a breaking change.
It does, however, break the above assertion, which may now fail like this:
Assert.Equal() Failure Expected: OK Actual: Created
You could attempt to fix this issue by instead verifying IsSuccessStatusCode:
Assert.True(response.IsSuccessStatusCode);
That permits the API to evolve, but introduces another problem.
Legibility #
What happens when an assertion against IsSuccessStatusCode fails?
Assert.True() Failure Expected: True Actual: False
That's not helpful. At least, you'd like to know which other status code was returned instead. You can address that concern by using an overload to Assert.True (most unit testing frameworks come with such an overload):
Assert.True( response.IsSuccessStatusCode, $"Actual status code: {response.StatusCode}.");
This produces more helpful error messages:
Actual status code: BadRequest. Expected: True Actual: False
Granted, the Expected: True, Actual: False lines are redundant, but at least the information you care about is there; in this example, the status code was 400 Bad Request.
Conciseness #
That's not bad. You can use that Assert.True overload everywhere you want to assert success. You can even copy and paste the code, because it's an idiom that doesn't change. Still, since I like to stay within 80 characters line width, this little phrase takes up three lines of code.
Surely, a helper method can address that problem:
internal static void AssertSuccess(this HttpResponseMessage response) { Assert.True( response.IsSuccessStatusCode, $"Actual status code: {response.StatusCode}."); }
I can now write a one-liner as an assertion:
response.AssertSuccess();
What's not to like?
Carrying coals to Newcastle #
If you're already familiar with the HttpResponseMessage class, you may have been reading this far with some frustration. It already comes with a method called EnsureSuccessStatusCode. Why not just use that instead?
response.EnsureSuccessStatusCode();
As long as the response status code is in the 200 range, this method call succeeds, but if not, it throws an exception:
Response status code does not indicate success: 400 (Bad Request).
That's exactly the information you need.
I admit that I can sometimes be too rigid in my thinking. I've known about this method for years, but I never thought of it as a unit testing assertion. It's as if there's a category in my programming brain that's labelled unit testing assertions, and it only contains methods that originate from unit testing.
I even knew about Debug.Assert, so I knew that assertions also exist outside of unit testing. That assertion isn't suitable for unit testing, however, so I never included it in my category of unit testing assertions.
The EnsureSuccessStatusCode method doesn't have assert in its name, but it behaves just like a unit testing assertion: it throws an exception with a useful error message if a criterion is not fulfilled. I just never thought of it as useful for unit testing purposes because it's part of 'production code'.
Conclusion #
The built-in EnsureSuccessStatusCode method is fine for unit testing purposes. Who knows, perhaps other class libraries contain similar assertion methods, although they may be hiding behind names that don't include assert. Perhaps your own production code contains such methods. If so, they can double as unit testing assertions.
I've been doing test-driven development for close to two decades now, and one of many consistent experiences is this:
- In the beginning of a project, the unit tests I write are verbose. This is natural, because I'm still getting to know the domain.
- After some time, I begin to notice patterns in my tests, so I refactor them. I introduce helper APIs in the test code.
- Finally, I realise that some of those unit testing APIs might actually be useful in the production code too, so I move them over there.
The code shown here is part of the sample code base that accompanies my book Code That Fits in Your Head.
Comments
Unfortunately, I can't agree that it's fine to use
EnsureSuccessStatusCodeor other similar methods for assertions. An exception of a kind different from assertion (XunitExceptionfor XUnit,AssertionExceptionfor NUnit) is handled in a bit different way by the testing framework.EnsureSuccessStatusCodemethod throwsHttpRequestException. Basically, a test that fails with such exception is considered as "Failed by error".For example, the test result message in case of 404 status produced by the
EnsureSuccessStatusCodemethod is:Where "System.Net.Http.HttpRequestException :" text looks redundant, as we do assertion. This also may make other people think that the reason for this failure is not an assertion, but an unexpected exception. In case of regular assertion exception type, that text is missing.
Other points are related to NUnit:
- Failure: a test assertion failed. (Status=Failed, Label=empty)
- Error: an unexpected exception occurred. (Status=Failed, Label=Error)
See NUnit Common Outcomes. Thus `EnsureSuccessStatusCode` produces "Error" status instead of desirable "Failure".EnsureSuccessStatusCodeas assertion inside multiple asserts."assertions"property gets into the test results XML file and might be useful. This property increments on assertion methods,EnsureSuccessStatusCode- obviously doesn't increment it.Yevgeniy, thank you for writing. This shows, I think, that all advice is contextual. It's been more than a decade since I regularly used NUnit, but I vaguely recall that it gives special treatment to its own assertion exceptions.
I also grant that the "System.Net.Http.HttpRequestException" part of the message is redundant. To make matters worse, even without that 'prologue' the information that we care about (the actual status code) is all the way to the right. It'll often require horizontal scrolling to see it.
That doesn't much bother me, though. I think that one should optimise for the most frequent scenario. For example, I favour taking a bit of extra time to write readable code, because the bottleneck in programming isn't typing. In the same vein, I find it a good trade-off being able to avoid adding more code against an occasional need to scroll horizontally. The cost of code isn't the time it takes to write it, but the maintenance burden it imposes, as well as the cognitive load it adds.
The point here is that while horizontal scrolling is inconvenient, the most frequent scenario is that tests don't fail. The inconvenience only manifests in the rare circumstance that a test fails.
I want to thank you, however, for pointing out the issues with NUnit. It reinforces the impression I already had of it, as a framework to stay away from. That design seems to me to impose a lock-in that prevents you from mixing NUnit with improved APIs.
In contrast, the xunit.core library doesn't include xunit.assert. I often take advantage of that when I write unit tests in F#. While the xUnit.net assertion library is adequate, Unquote is much better for F#. I appreciate that xUnit.net enables me to combine the tools that work best in a given situation.
I see your point, Mark. Thanks for your reply.
I would like to add a few words in defense of the NUnit.
First of all, there is no problem to use an extra library for NUnit assertions too. For example, I like FluentAssertions that works great for both XUnit and NUnit.
NUnit 3 also has a set of useful features that are missing in XUnit like:
I can recommend to reconsider the view of NUnit, as the library progressed well during its 3rd version. XUnit and NUnit seem interchangeable for unit testing. But NUnit, from my point of view, provides the features that are helpful in integration and UI testing.
Yevgeniy, I haven't done UI testing since sometime in 2005, I think, so I wasn't aware of that. It's good to know that NUnit may excel at that, and the other scenarios you point out.
To be clear, while I still think I may prefer xUnit.net, I don't consider it perfect.
I would like to mention FluentAssertions.Web, a library I wrote with the purpose to solve exact this problem I had while maintaining API tests. It's frustrating not to tell in two seconds why a test failed. Yeah, it is a Bad Request, but why. Maybe the response is in the respons content?
Wouldn't be an idea to see the response content? This FluentAssertions extension outputs everything starting from the request to the response, when the test fails.Adrian, thank you for writing. Your comment inspired me to write an article on when tests fail. This article is now in the publication queue. I'll try to remember to come back here and add a link once it's published.
(Placeholder)