Self-hosted integration tests in ASP.NET

A way to self-host a REST API and test it through HTTP.

In 2020 I developed a sizeable code base for an online restaurant REST API. In the spirit of outside-in TDD, I found it best to test the HTTP behaviour of the API by actually interacting with it via HTTP.

Sometimes ASP.NET offers more than one way to achieve the same end result. For example, to return 200 OK, you can use both OkObjectResult and ObjectResult. I don't want my tests to be coupled to such implementation details, so by testing an API via HTTP instead of using the ASP.NET object model, I decouple the two.

You can easily self-host an ASP.NET web API and test it using an HttpClient. In this article, I'll show you how I went about it.

The code shown here is part of the sample code base that accompanies my book Code That Fits in Your Head.

Reserving a table #

In true outside-in fashion, I'll first show you the test. Then I'll break it down to show you how it works.

[Fact]
public async Task ReserveTableAtNono()
{
    using var api = new SelfHostedApi();
    var client = api.CreateClient();
    var at = DateTime.Today.AddDays(434).At(20, 15);
    var dto = Some.Reservation.WithDate(at).WithQuantity(6).ToDto();
 
    var response = await client.PostReservation("Nono", dto);
 
    await AssertRemainingCapacity(client, at, "Nono", 4);
    await AssertRemainingCapacity(client, at, "Hipgnosta", 10);
}

This test uses xUnit.net 2.4.1 to make a reservation at the restaurant named Nono. The first line that creates the api variable spins up a self-hosted instance of the REST API. The next line creates an HttpClient configured to communicate with the self-hosted instance.

The test proceeds to create a Data Transfer Object that it posts to the Nono restaurant. It then asserts that the remaining capacity at the Nono and Hipgnosta restaurants are as expected.

You'll see the implementation details soon, but I first want to discuss this high-level test. As is the case with most of my code, it's far from perfect. If you're not familiar with this code base, you may have plenty of questions:

Why does it make a reservation 434 days in the future? Why not 433, or 211, or 1?
Is there anything significant about the quantity 6?
Why is the expected remaining capacity at Nono 4?
Why is the expected remaining capacity at Hipgnosta 10? Why does it even verify that?

To answer the easiest question first: There's nothing special about 434 days. The only requirement is that it's a positive number, so that the reservation is in the future. That makes this test a great candidate for a property-based test.

The three other questions are all related. A bit of background is in order. I wrote this test during a process where I turned the system into a multi-tenant system. Before that change, there was only one restaurant, which was Hipgnosta. I wanted to verify that if you make a reservation at another restaurant (here, Nono) it changes the observable state of that restaurant, and not of Hipgnosta.

The way these two restaurants are configured, Hipgnosta has a single communal table that seats ten guests. This explains why the expected capacity of Hipgnosta is 10. Making a reservation at Nono shouldn't affect Hipgnosta.

Nono has a more complex table configuration. It has both standard and communal tables, but the largest table is a six-person communal table. There's only one table of that size. The next-largest tables are four-person tables. Thus, a reservation for six people reserves the largest table that day, after which only four-person and two-person tables are available. Therefore the remaining capacity ought to be 4.

The above test knows all this. You are welcome to criticise such hard-coded knowledge. There's a real risk that it might make it more difficult to maintain the test suite in the future.

Certainly, had this been a unit test, and not an integration test, I wouldn't have accepted so much implicit knowledge - particularly because I mostly apply functional architecture, and pure functions should have isolation. Functions shouldn't depend on implicit global state; they should return a value based on input arguments. That's a bit of digression, though.

These are integration tests, which I mostly use for smoke tests and to verify HTTP-specific behaviour. I have unit tests for fine-grained testing of edge cases and variations of input. While I wouldn't accept so much implicit knowledge from a unit test, I find that it so far works well with integration tests.

Self-hosting #

It only takes a WebApplicationFactory to self-host an ASP.NET API. You can use it directly, but if you want to modify the hosted service in some way, you can also inherit from it.

I want my self-hosted integration tests to run as state-based tests that use an in-memory database instead of SQL Server. I've defined SelfHostedApi for that purpose:

internal sealed class SelfHostedApi : WebApplicationFactory<Startup>
{
    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureServices(services =>
        {
            services.RemoveAll<IReservationsRepository>();
            services.AddSingleton<IReservationsRepository>(new FakeDatabase());
        });
    }
}

The way that WebApplicationFactory works, its ConfigureWebHost method runs after the Startup class' ConfigureServices method. Thus, when ConfigureWebHost runs, the services collection is already configured to use SQL Server. As Julius H so kindly pointed out to me, the RemoveAll extension method removes all existing registrations of a service. I use it to remove the SQL Server dependency from the system, after which I replace it with a test-specific in-memory implementation.

Since the in-memory database is configured with Singleton lifetime, that instance is going to be around for the lifetime of the service. While it's only keeping track of things in memory, it'll keep state until the service shuts down, which happens when the above api variable goes out of scope.

Notice that I named the class by the role it plays rather than which base class it derives from.

Posting a reservation #

The PostReservation method is an extension method on HttpClient:

internal static async Task<HttpResponseMessage> PostReservation(
    this HttpClient client,
    string name,
    object reservation)
{
    string json = JsonSerializer.Serialize(reservation);
    using var content = new StringContent(json);
    content.Headers.ContentType.MediaType = "application/json";
 
    var resp = await client.GetRestaurant(name);
    resp.EnsureSuccessStatusCode();
    var rest = await resp.ParseJsonContent<RestaurantDto>();
    var address = rest.Links.FindAddress("urn:reservations");
 
    return await client.PostAsync(address, content);
}

It's part of a larger set of methods that enables an HttpClient to interact with the REST API. Three of those methods are visible here: GetRestaurant, ParseJsonContent, and FindAddress. These, and many other, methods form a client API for interacting with the REST API. While this is currently test code, it's ripe for being extracted to a reusable client SDK library.

I'm not going to show all of them, but here's GetRestaurant to give you a sense of what's going on:

internal static async Task<HttpResponseMessage> GetRestaurant(this HttpClient client, string name)
{
    var homeResponse = await client.GetAsync(new Uri("", UriKind.Relative));
    homeResponse.EnsureSuccessStatusCode();
    var homeRepresentation = await homeResponse.ParseJsonContent<HomeDto>();
    var restaurant = homeRepresentation.Restaurants.First(r => r.Name == name);
    var address = restaurant.Links.FindAddress("urn:restaurant");
 
    return await client.GetAsync(address);
}

The REST API has only a single documented address, which is the 'home' resource at the relative URL ""; i.e. the root of the API. In this incarnation of the API, the home resource responds with a JSON array of restaurants. The GetRestaurant method finds the restaurant with the desired name and finds its address. It then issues another GET request against that address, and returns the response.

Verifying state #

The verification phase of the above test calls a private helper method:

private static async Task AssertRemainingCapacity(
    HttpClient client,
    DateTime date,
    string name,
    int expected)
{
    var response = await client.GetDay(name, date.Year, date.Month, date.Day);
    var day = await response.ParseJsonContent<CalendarDto>();
    Assert.All(
        day.Days.Single().Entries,
        e => Assert.Equal(expected, e.MaximumPartySize));
}

It uses another of the above-mentioned client API extension methods, GetDay, to inspect the REST API's calendar entry for the restaurant and day in question. Each calendar contains a series of time entries that lists the largest party size the restaurant can accept at that time slot. The two restaurants in question only have single seatings, so once you've booked a six-person table, you have it for the entire evening.

Notice that verification is done by interacting with the system itself. No Back Door Manipulation is required. I favour this if at all possible, since I believe that it offers better confidence that the system behaves as it should.

Conclusion #

It's been possible to self-host .NET REST APIs for testing purposes at least since 2012, but it's only become easier over the years. All you need to get started is the WebApplicationFactory<TEntryPoint> class, although you're probably going to need a derived class to override some of the system configuration.

From there, you can interact with the self-hosted system using the standard HttpClient class.

Since I configure these tests to run on an in-memory database, the execution time is comparable to 'normal' unit tests. I admit that I haven't measured it, but that's because I haven't felt the need to do so.

Comments

Grzegorz Gałęzowski #

Hi Mark, 2 questions from me:

What kind of integration does this test verify? The integration between the HTTP request processing part and the service logic? If so, why fake the database only and not the whole logic?
If you fake the database here, then would you have a separate test for testing integration with DB (e.g. some kind of "adapter test", as described in the GOOS book)?

2021-01-25 20:34 UTC

Mark Seemann #

Grzegorz, thank you for writing.

1. As I already hinted at, the motivation for this test was that I was expanding the code from a single-tenant to a multi-tenant system. I did this in small steps, using tests to drive the augmented behaviour. Before I added the above test, I'd introduced the concept of a restaurant ID to identify each restaurant, and initially added a method overload to my data access interface that required such an ID:

Task Create(int restaurantId, Reservation reservation);

I was using the Strangler pattern to be able to move in small steps. One step was to add overloads like the above. Subsequent steps would be to move existing callers from the 'legacy' overload to the new overload, and then finally delete the 'legacy' overload.

When moving from a single-tenant system to a multi-tenant system, I had to grandfather in the existing restaurant, which I arbitrarily chose to give the restaurant ID 1. During this process, I kept the existing system working so as to not break existing clients.

I gradually introduced method overloads that took a restaurant ID as a parameter (as above), and then deleted the legacy methods that didn't take a restaurant ID. In order to not break anything, I'd initially be calling the new methods with the hard-coded restaurant ID 1.

The above test was one in a series of tests that followed. Their purpose was to verify that the system had become truly multi-tenant. Before I added that test, an attempt to make a reservation at Nono would result in a call to Create(1, reservation) because there was still hard-coded restaurant IDs left in the code base.

(To be honest, the above is a simplified account of events. It was actually more complex than that, since it involved hypermedia controls. I also, for reasons that I may divulge in the future, didn't perform this work behind a feature flag, which under other circumstances would have been appropriate.)

What the above test verifies, then, is that the reservation gets associated with the correct restaurant, instead of the restaurant that was grandfathered in (Hipgnosta). An in-memory (Fake) database was sufficient to demonstrate that.

Why didn't I use the real, relational database? Read on.

2. As I've recently discussed, integration tests that involve SQL Server are certainly possible. They tend to be orders-of-magnitudes slower than unit tests that run entirely in memory, so I only add them if I deem them necessary.

I interact with databases according to the Dependency Inversion Principle. The above Create method is an example of that. The method is defined by the needs of the client code rather than on implementation details.

The actual implementation of the data access interface is, as you imply, an Adapter. It adapts the SQL Server SDK (i.e. ADO.NET) to my data access interface. As long as I can keep such Adapters Humble Objects I don't cover them by tests.

Such Adapters are often just mappings: This class field maps to that table column, that field to that other column, and so on. If there's no logic, there isn't much to test.

This doesn't mean that bugs can't appear in database Adapters. If that happens, I may introduce regression tests that involve the database. I prefer to do this in a separate test suite, however, since such tests tend to be slow, and TDD test suites should be fast.

2021-01-26 8:32 UTC

Brian Elgaard Bennett #

Hi Mark, I have been a fan of "integration" style testing since I found Andrew Locks's post on the topic. In fact, I personally find that it makes sense to take that idea to one extreme.

As far as I can see, your test assumes something like "Given that no reservations have been made". Did you consider to make that "Given" explicitly recognizable in your test?

I am asking because this is a major pet topic of mine and I would love to hear your thoughts. I am not suggesting writing Gherkin instead of C# code, just if and how you would make the "Given" assumption explicit.

A more practical comment is on your use of services.RemoveAll<IReservationsRepository>(); It seems extreme to remove all, as all you want to achieve is to overwrite a few service declarations. Andrew Lock demonstrates how to keep all the ASP.NET middleware intact by keeping most of the service registrations.

I usually use ConfigureTestServices on WebApplicationFactory, something like,

factory.WithWebHostBuilder(builder => builder.ConfigureTestServices(MyTestCompositionRoot.Initialize));

2021-02-22 10:40 UTC

Mark Seemann #

Brian, thank you for writing. You're correct that one implicit assumption is that no prior reservations have been made to Nono on that date. The reality is, as you suggest, that there are no reservations at all, but the test doesn't require that. It only relies on the weaker assumption that there are prior reservations to neither Nono nor Hipgnosta on the date in question.

I often find it tricky to make 'negative' assumptions explicit. How much should one highlight? Is it also relevant to highlight that SelfHostedApi isn't going to send notification emails? That it doesn't write to a persistent log? That it doesn't have the capability to launch missiles?

I'm not saying that it's irrelevant to highlight certain assumptions, but I'm not sure that I have a clear heuristic for it. After all, it depends on what the reader cares about, and how clear the assumptions are.

If it became clear to me that the assumption of 'no prior reservations' was important to the reader, I might consider to embed that in the name of the test, or perhaps by adding a comment. Another option is to hide the initialisation of SelfHostedApi behind a required factory method so that you'd have to initialise it as using var api = SelfHostedApi.StartEmpty();

In this particular code base, though, that SelfHostedApi is used repeatedly. A programmer who regularly works with that code base will quickly internalise the assumptions embedded in that self-hosted service.

Granted, it can be a dangerous strategy to rely too much on implicit assumptions, because it can make it more difficult to change things if those assumptions change in the future. I do, after all, favour the wisdom from the Zen of Python: Explicit is better than implicit.

Making such things as we discuss here explicit does, however, tend to make the tests more verbose. Pragmatically, then, there's a balance to strike. I'm not claiming that the test shown here is perfect. In a a living code base, this would be a concern that I would keep a constant eye on. I'd fiddle with it, trying out making things more explicit, less explicit, and so on, to see what works best in context.

Your other point about services.RemoveAll<IReservationsRepository>() I don't understand. Why is that extreme? It's literally the most minimal, precise change I can make. It removes only the IReservationsRepository, of which there's exactly one Singleton instance:

var connStr = Configuration.GetConnectionString("Restaurant");
services.AddSingleton<IReservationsRepository>(sp =>
{
    var logger = sp.GetService<ILogger<LoggingReservationsRepository>>();
    var postOffice = sp.GetService<IPostOffice>();
    return new EmailingReservationsRepository(
        postOffice,
        new LoggingReservationsRepository(
            logger,
            new SqlReservationsRepository(connStr)));
});

The integration test throws away that Singleton object graph, but nothing else.

2021-02-23 9:38 UTC

Brian Elgaard Bennett #

Hi Mark, thanks for a thorough answer.

Alas, my second question was not thought through - I somehow thought you threw away all service registrations, which you obviously do not. In fact, I do exactly what you do, usually with Replace and thereby implicitly assume that there is only one registration. My Bad.

We very much agree in the importance of favouring explicit assumptions.

The solution to stating 'negative' assumptions, as long as we talk about the 'initial context' (the 'Given') is actually quite simple if your readers know that you always state assumptions explicitly. If none are stated, it's the same as saying 'the world is empty' - at least when it comes to the bounded context of whatever you are testing. I find that the process of identifying a minimal set of assumptions per test is a worthwhile exercise.

So, IMHO it boils down to consistently stating all assumptions and making the relevant bounded context clear.

2021-02-23 12:18 UTC

Mark Seemann #

Brian, thank you for writing. I agree that it's a good rule of thumb to expect the world to be empty unless explicitly stated otherwise. This is the reason that I love functional programming.

One question, however, is whether a statement is sufficiently explicit. The integration test shown here is a good example. What's actually happening is that SelfHostedApi sets up a self-hosted service configured in a certain way. It contains three restaurants, of which two are relevant in this particular test. Each restaurant comes with quite a bit of configuration: opening time, closing time, seating duration, and the configuration of tables. Just consider the configuration of Nono:

{
  "Id": 2112,
  "Name": "Nono",
  "OpensAt": "18:00",
  "LastSeating": "21:00",
  "SeatingDuration": "6:00",
  "Tables": [
    {
      "TableType": "Communal",
      "Seats": 6
    },
    {
      "TableType": "Communal",
      "Seats": 4
    },
    {
      "TableType": "Standard",
      "Seats": 2
    },
    {
      "TableType": "Standard",
      "Seats": 2
    },
    {
      "TableType": "Standard",
      "Seats": 4
    },
    {
      "TableType": "Standard",
      "Seats": 4
    }
  ]
}

The code base also enables a test writer to express the above configuration as code instead of JSON, but the point I'm trying to make is that you probably don't want that amount of detail to appear in the Arrange phase of each test, regardless of how explicit it'd be.

To work around an issue like this, you might instead define an immutable test data variable called nono. Then, in each test that uses these restaurants, you might include something like AddRestaurant(nono) or AddRestaurant(hipgnosta) in the Arrange phase (hypothetical API). That's not quite as explicit, because it hides the actual values away, but is probably still good enough.

Then you find yourself always adding the same restaurants, so you say to yourself: It'd be nice if I had a helper method like AddStandardRestaurants. So, you may add such a helper method.

But you find that for the integration tests, you always call that helper method, so ultimately, you decide to just roll it into the definition of SelfHostedApi. That's basically what's going on here.

I don't recall whether I explicitly stated the following in the article, but I use integration tests consistent with the test pyramid. The purpose of the integration tests is to verify that components integrate correctly, and that the HTTP API behaves according to contract. In my experience, I can let the 'givens' be somewhat implicit and still keep the code maintainable. For integration tests, that is.

On the unit level, I favour pure functions, which are intrinsically testable. Due to isolation, everything that affects the behaviour of a pure function must be explicitly passed as arguments, and as a bi-product, they become explicit as part of a test's Arrange phase.

But, again, let me reiterate that I agree with you. I don't claim that the code shown here is perfect. It's so implicit that it makes me uncomfortable too, but on the other hand, I think that a test like the above communicates intent well. The more explicit details one adds, the more they may drown out the intent. It's a difficult balance to strike.

2021-02-25 12:34 UTC

Brian Elgaard Bennett #

Hi Mark, I'm happy that we agree. Still, this is a pet topic of mine so please allow me one additional comment.

I wish all code had at least a functional core, as that would make testing so much easier. You could say that explicitly stating initial context resembles testing pure functions.

I do believe that it is possible to state initial context without being overwhelmed by details. It can by no means be fully automated - this is where the mind of a tester must be used. Also, it will never be perfect, but at least it can be documented somewhat systematically and kept under version control, so the reasoning can be challenged.

In your example, I would not add Nono or Hipgnosta or a standard restaurant, as those names say noting about what makes my test pass.

Rather, I would add e.g. a "restaurant with a 6 and a 4 person table", as this seems to be what makes your test pass, if I understood it correctly. In another test I might have "restaurant which is open from 6 PM with last seating at 9 PM" if I want to test time aspects of booking. It may be the same Json file used in those two tests, because what's most important is that you document what makes each individual test pass.

However, if it's not too much trouble (it sometimes is) I would even try to minimize the initial context. For example, I would test timing aspects with a minimal restaurant with a single table. Unless, timing and tables interact - maybe late bookings are OK if the restaurant is starved for customers? This is where the "tester mind" comes in handy.

In your restaurant example, that would mean having quite a few restaurant definitions, but only definitions which include the needed context for each test. My experience is that minimizing the initial context will teach you a lot about the code. This is similar to unit tests which focus on a small part of code, but "minimizing initial context" allows for less fragile tests than when "minimizing code".

We started using this approach because our tests did not give sufficient confidence. Tests were flaky and it was difficult to see what was tested and why. I can safely say that this is much better now.

My examples of initial context above may appear like they could grow out of control, but our experience so far is that it is possible to find a relatively small set of equivalence classes of initial context items, even for fairly complex functionality.

2021-02-28 15:24 UTC

Mark Seemann #

Brian, I agree that in unit testing, being able to reduce test cases to minimal examples is useful. Once you have those minimal examples, being explicit about all preconditions is more attainable. This is why, when I coach teams, I try to teach them about equivalence class partitioning. If, within each equivalence class, you can impose some sort of (partial) ordering, you may be able to pick a canonical representation that constitutes the minimal example.

This is basically what the shrinking process of property-based testing frameworks attempt to do.

Thus, when my testing goal is to explore the boundaries of the system under test, I go to great lengths to ensure that the arrange phase is as explicit as possible. I recently published an example of how I use property-based testing to such an end.

The goal of the integration test in the present article, however, is not to explore boundary cases or verify the correctness of algorithms in question. I have unit tests for that. The goal is to examine whether things may be correctly 'clicked together'.

Expressing an explicit minimal context for the above test is more involved than you outline. Having two restaurants, with two differently sized tables, is only the beginning. You must also ensure that they are open at the same time, so that an attempted reservation would fit both. Furthermore, you must then pick the date and time of the reservation to fit within that time window. This is still not hopelessly difficult, and might make for a good exercise, but it could easily add 4-5 extra lines of code to the test.

And again: had this been a unit test, I'd happily added those extra lines, but for an integration test, I felt that it was less important.

2021-03-02 11:24 UTC

Published: Monday, 25 January 2021 07:45:00 UTC

Self-hosted integration tests in ASP.NET by Mark Seemann

Reserving a table #

Self-hosting #

Posting a reservation #

Verifying state #

Conclusion #

Comments

Wish to comment?

Published

Tags