Using Polly with F# async workflows

Tuesday, 30 May 2017 12:03:00 UTC

How to use Polly as a Circuit Breaker in F# async workflows.

Release It! describes a stability design pattern called Circuit Breaker, which is used to fail fast if a downstream service is experiencing problems.

I recently had to add a Circuit Breaker to an F# async workflow, and although Circuit Breaker isn't that difficult to implement (my book contains an example in C#), I found it most prudent to use an existing implementation. Polly seemed a good choice.

In my F# code base, I was already working with an 'abstraction' of the type HttpRequestMessage -> Async<HttpResponseMessage>: given an HttpClient called client, the implementation is as simple as client.SendAsync >> Async.AwaitTask. Since SendAsync can throw HttpRequestException or WebException, I wanted to define a Circuit Breaker policy for these two exception types.

While Polly supports policies for Task-based APIs, it doesn't automatically work with F# async workflows. The problem is that whenever you convert an async workflow into a Task (using Async.AwaitTask), or a Task into an async workflow (using Async.StartAsTask), any exceptions thrown will end up buried within an AggregateException. In order to dig them out again, I first had to write this function:

// Exception -> bool
let rec private isNestedHttpException (e : Exception) =
    match e with
    | :? AggregateException as ae ->
        ae.InnerException :: Seq.toList ae.InnerExceptions
        |> Seq.exists isNestedHttpException
    | :? HttpRequestException -> true
    | :? WebException -> true
    | _ -> false

This function recursively searches through all inner exceptions of an AggregateException and returns true if it finds one of the exception types I'm interested in handling; otherwise, it returns false.

This predicate enabled me to write the Polly policy I needed:

open Polly
// int -> TimeSpan -> CircuitBreaker.CircuitBreakerPolicy
let createPolicy exceptionsAllowedBeforeBreaking durationOfBreak =
        .Handle<AggregateException>(fun ae -> isNestedHttpException ae)
        .CircuitBreakerAsync(exceptionsAllowedBeforeBreaking, durationOfBreak)

Since Polly exposes an object-oriented API, I wanted a curried alternative, so I also wrote this curried helper function:

// Policy -> ('a -> Async<'b>) -> 'a -> Async<'b>
let private execute (policy : Policyf  req =
    policy.ExecuteAsync(fun () -> f req |> Async.StartAsTask) |> Async.AwaitTask

The execute function executes any function of the type 'a -> Async<'b> with a Polly policy. As you can see, there's some back-and-forth between Tasks and async workflows, so this is probably not the most efficient Circuit Breaker ever configured, but I wagered that since the underlying operation was going to involve an HTTP request and response, the overhead would be insignificant. No one has complained yet.

When Polly opens the Circuit Breaker, it throws an exception of the type BrokenCircuitException. Again because of all the marshalling, this also gets wrapped within an AggregateException, so I had to write another function to unwrap it:

// Exception -> bool
let rec private isNestedCircuitBreakerException (e : Exception) =
    match e with
    | :? AggregateException as ae ->
        ae.InnerException :: Seq.toList ae.InnerExceptions
        |> Seq.exists isNestedCircuitBreakerException
    | :? CircuitBreaker.BrokenCircuitException -> true
    | _ -> false

The isNestedCircuitBreakerException is similar to isNestedHttpException, so it'd be tempting to refactor. I decided, however, to rely on the rule of three and leave both functions as they were.

In my F# code I prefer to handle application errors using Either values instead of relying on exceptions, so I wanted to translate any BrokenCircuitException to a particular application error. With the above isNestedCircuitBreakerException predicate, this was now possible with a try/with expression:

// Policy -> ('a -> Async<'b>) -> 'a -> Async<Result<'b, BoundaryFailure>>
let sendUsingPolicy policy send req =
    async {
            let! resp = req |> execute policy send
            return Result.succeed resp
        with e when isNestedCircuitBreakerException e ->
                return Controllers.StabilityFailure }

This function takes a policy, a send function that actually sends the request, and the request to send. If all goes well, the response is lifted into a Success case and returned. If the Circuit Breaker is open, a StabilityFailure value is returned instead.

Since the with expression uses an exception filter, all other exceptions will still be thrown from this function.

It might still be worthwhile to look into options for a more F#-friendly Circuit Breaker. One option would be to work with the Polly maintainers to add such an API to Polly itself. Another option would be to write a separate F# implementation.


Johannes Egger
Nice post, thanks.
There are two minor points I want to address: * I think instead of the recursive search for a nested exception you can use AggregateException.Flatten() |> Seq.exists ...
* And I know that you know that a major difference between Async and Task is that a Task is typically started whereas an Async is not. So it might be irritating that calling execute already starts the execution. If you wrapped the body of execute inside another async block it would be lazy as usual, I think.
2017-06-03 21:21 UTC

Simple holidays

Monday, 24 April 2017 13:42:00 UTC

A story about arriving at the simplest solution that could possibly work.

The Zen of Python states: Simple is better than complex. If I've met a programmer who disagrees with that, I'm not aware of it. It's hardly a controversial assertion, but what does 'simplicity' mean? Can you even identify a simple solution?

I often see software developers defaulting to complex solutions, because a simpler solution isn't immediately obvious. In retrospect, a simple solution often is obvious, but only once you've found it. Until then, it's elusive.

I'd like to share a story in which I arrived at a simple solution after several false starts. I hope it can be an inspiration.

Dutch holidays #

Recently, I had to write some code that takes into account Dutch holidays. (In order to address any confusion that could arise from this: No, I'm not Dutch, I'm Danish, but currently, I'm doing some work on a system targeting a market in the Netherlands.) Specifically, given a date, I had to find the latest possible Dutch bank day on or before that date.

For normal week days (Monday to Friday), it's easy, because such a date is already a bank date. In other words, in that case, you can simply return the input date. Also, in normal weeks, given a Saturday or Sunday, you should return the preceding Friday. The problem is, however, that some Fridays are holidays, and therefore not bank days.

Like many other countries, the Netherlands have complicated rules for determining official holidays. Here are some tricky parts:

  • Some holidays always fall on the same date. One example is Bevrijdingsdag (Liberation Day), which always falls on May 5. This holiday celebrates a historic event (the end of World War II in Europe), so if you wanted to calculate bank holidays in the past, you'd have to figure out in which year this became a public holiday. Surely, at least, it must have been 1945 or later.
  • Some holidays fall on specific week days. The most prominent example is Easter, where Goede Vrijdag (Good Friday) always (as the name implies) falls on a Friday. Which Friday exactly can be calculated using a complicated algorithm.
  • One holiday (Koningsdag (King's Day)) celebrates the king's birthday. The date is determined by the currently reigning monarch's birthday, and it's called Queen's Day when the reigning monarch is a queen. Obviously, the exact date changes depending on who's king or queen, and you can't predict when it's going to change. And what will happen if the current monarch abdicates or dies before his or her birthday, but after the new monarch's birthday? Does that mean that there will be no such holiday that year? Or what about the converse? Could there be two such holidays if a monarch abdicates after his or her birthday, and the new monarch's birthday falls later the same year?
Such problems aren't particular to the Netherlands. In Denmark, we can find similar examples, as I think you can do in many other countries. Ultimately, what constitutes an official holiday is a political decision.

Figuring out if a date is a bank day, then, is what you might call an 'interesting' problem. How would you solve it? Before you read on, take a moment to consider how you'd attempt to solve the problem. If you will, you can consider the test cases immediately below to get a better sense of the problem.

Test cases #

Here's a small set of test cases that I wrote in order to describe the problem:

Test case Input date Expected output
Good Friday2017-04-142017-04-13
Saturday after Good Friday2017-04-152017-04-13
Sunday after Good Friday2017-04-162017-04-13
Easter Monday2017-04-172017-04-13
Ascension Day - Thursday2017-05-252017-05-24
Whit Monday2110-05-262110-05-23
Liberation Day9713-05-059713-05-04
You'll notice that while I wrote most of my test cases in the near future (they're actually already in the past, now), I also added some far future dates for good measure. This assumes that the Netherlands will still celebrate Christian religious holidays in a hundred years from now, or their liberation day in 9713. That the Netherlands still exist then as the country we know today is a more than dubious assumption.

Option: query a third-party service #

How would you solve the problem? The first solution that occurred to me was to use a third-party service. My guess is that most developers would consider this option. After all, it's essentially third-part data. The official holidays are determined by a third party, in this case the Dutch state. Surely, some Dutch official organisation would publish the list of official holidays somewhere. Perhaps, if you're lucky, there's even an on-line service you can query in order to download the list of holidays in some machine-readable format.

There are, however, problems with this alternative: if you query such a service each time you need to find an appropriate bank date, how are you going to handle network errors? What if the third-part service is (temporarily) unavailable?

Since I'm trying to figure out bank dates, you may already have guessed that I'm handling money, so it's not desirable to simple throw an exception and say that a caller would have to try again later. This could lead to loss of revenue.

Querying a third-party service every time you need to figure out a Dutch bank holiday is out of the question for that reason. It's also likely to be inefficient.

Option: cache third-party data #

Public holidays rarely change, so your next attempt could be a variation of the previous. Use third-party data, but instead of querying a third-party service every time you need the information, cache it.

The problem with caching is that you're not guaranteed that the data you seek is in the cache. At application start, caches are usually empty. You'd have to rely on making one good query to the third-party data source in order to put the data in the cache. Only if that succeeds can you use the cache. This, again, leaves you vulnerable to the normal failure modes of distributed computing. If you can't reach the third-party data source, you have nothing to put in the cache.

This can be a problem at application start, or when the cache data expires.

Using a cache reduces the risk that the data is unavailable, but it doesn't eliminate it. It also adds complexity in the form of a cache that has to be configured and managed. Granted, you can use a reusable cache library or service to minimise that cost, so it may not be a big deal. Still, when making a decision about application architecture, I think it helps to explicitly identify advantages and disadvantages.

Using a cache felt better to me, but I still wasn't happy. Too many things could go wrong.

Option: persistent cache #

An incremental improvement on the previous option would be to write the cache data to persistent storage. This takes care of the issue with the cache being empty at application start-up. You can even deal with cache expiry by keep using stale data if you can't reach the 'official' source of the data.

It leaves me a bit concerned, though, because if you allow the system to continue working with stale data, perhaps the application could enter a state where the data never updates. This could happen if the official data source moves, or changes format. In such a case, your application would keep trying to refresh the cache, and it would permanently fail. It would permanently run with stale data. Would you ever discover that problem?

My concern is that the application could silently fail. You could counter that by logging a warning somewhere, but that would introduce a permanent burden on the team responsible for operating the application. This isn't impossible, but it does constitute an extra complexity. This alternative still didn't feel good to me.

Option: cron #

Because I wasn't happy with any of the above alternatives, I started looking for different approaches to the problem. For a short while, I considered using a .NET implementation of cron, with a crontab file. As far as I can tell, though there's no easy way to define Easter using cron, so I quickly abandoned that line of inquiry.

Option: Nager.Date #

I wasn't entirely done with idea of calculating holidays on the fly. While calculating Easter is complicated, it can be done; there is a well-defined algorithm for it. Whenever I run into a general problem like this, I assume that someone has already done the required work, and this is also the case here. I quickly found an open source library called Nager.Date; I'm sure that there are other alternatives, but Nager.Date looks like it's more than good enough.

Such a library would be able to calculate all holidays for a given year, based on the algorithms embedded in it. That looked really promising.

And yet... again, I was concerned. Official holidays are, as we've already established, politically decided. Using an algorithmic approach is fundamentally wrong, because that's not really how the holidays are determined. Holidays are defined by decree; it just so happens that some of the decrees take the form of an algorithm (such as Easter).

What would happen if the Dutch state decides to add a new holiday? Or to take one away? Of when a new monarch is crowned? In order to handle such changes, we'd now have to hope that Nager.Date would be updated. We could try to make that more likely to happen by sending a pull request, but we'd still be vulnerable to a third party. What if the maintainer of Nager.Date is on vacation?

Even if you can get a change into a library like Nager.Date, how is the algorithmic approach going to deal with historic dates? If the monarch changes, you can update the library, but does it correctly handle dates in the past, where the King's Day was different?

Using an algorithm to determine a holiday seemed promising, but ultimately, I decided that I didn't like this option either.

Option: configuration file #

My main concern about using an algorithm is that it'd make it difficult to handle arbitrary changes and exceptional cases. If we'd use a configuration file, on the other hand, we could always edit the configuration file in order to add or remove holidays for a given year.

In essence, I was envisioning a configuration file that simply contained a list of holidays for each year.

That sounds fairly simple and maintainable, but from where should the data come?

You could probably download a list of official holidays for the next few years, like 2017, 2018, 2019, and so on, but the list would be finite, and probably not cover more than a few years into the future.

What if, for example, I'd only be able to find an official list that goes to 2020? What will happen, then, when our application enters 2021? To the rest of the code base, it'd look like there were no holidays in 2021.

At this time we can expect that new official lists have been published, so a programmer could obtain such a list and update the configuration file when it's time. This, unfortunately, is easy to forget. Four years in the future, perhaps none of the original programmers are left. It's more than likely that no one will remember to do this.

Option: algorithm-generated configuration file #

The problem that the configuration data could 'run out' can be addressed by initialising the configuration file with data generated algorithmically. You could, for example, ask Nager.Date to generate all the holidays for the next many years. In fact, the year 9999 is the maximum year handled by .NET's System.DateTime, so you could ask it to generate all the holidays until 9999.

That sounds like a lot, but it's only about half a megabyte of data...

This solves the problem of 'running out' of holiday data, but still enables you to edit the holiday data when it changes in the future. For example, if the King's Day changes in 2031, you can change all the King's Day values from 2031 onward, while retaining the correct values for the previous years.

This seems promising...

Option: hard-coded holidays #

I almost decided to use the previous, configuration-based solution, and I was getting ready to invent a configuration file format, and a reader for it, and so on. Then I recalled Mike Hadlow's article about the configuration complexity clock.

I'm fairly certain that the only people who would be editing a hypothetical holiday configuration file would be programmers. In that case, why put the configuration in a proprietary format? Why deal with the hassle of reading and parsing such a file? Why not put the data in code?

That's what I decided to do.

It's not a perfect solution. It's still necessary to go and change that code file when the holiday rules change. For example, when the King's Day changes, you'd have to edit the file.

Still, it's the simplest solution I could come up with. It has no moving parts, and uses a 'best effort' approach in order to guarantee that holidays will always be present. If you can come up with a better alternative, please leave a comment.

Data generation #

Nager.Date seemed useful for generating the initial set of holidays, so I wrote a small F# script that generated the necessary C# code snippets:

#r @"packages/Nager.Date.1.3.0/lib/net45/Nager.Date.dll"
open System.IO
open Nager.Date
let formatHoliday (h : Model.PublicHoliday) =
    let y, m, d = h.Date.Year, h.Date.Month, h.Date.Day
    sprintf "new DateTime(%i%2i%2i), // %s/%s" y m d h.Name h.LocalName
let holidays =
    |> Seq.collect (fun y -> DateSystem.GetPublicHoliday (CountryCode.NL, y))
    |> formatHoliday
File.WriteAllLines (__SOURCE_DIRECTORY__ + "/dutch-holidays.txt", holidays)

This script simply asks Nager.Date to calculate all Dutch holidays for the years 2017 to 9999, format them as C# code snippets, and write the lines to a text file. The size of that file is 4 MB, because the auto-generated code comments also take up some space.

First implementation attempt #

The next thing I did was to copy the text from dutch-holidays.txt to a C# code file, which I had already prepared with a class and a few methods that would query my generated data. The result looked like this:

public static class DateTimeExtensions
    public static DateTimeOffset AdjustToLatestPrecedingDutchBankDay(
        this DateTimeOffset value)
        var candidate = value;
        while (!(IsDutchBankDay(candidate.DateTime)))
            candidate = candidate.AddDays(-1);
        return candidate;
    private static bool IsDutchBankDay(DateTime date)
        if (date.DayOfWeek == DayOfWeek.Saturday)
            return false;
        if (date.DayOfWeek == DayOfWeek.Sunday)
            return false;
        if (dutchHolidays.Contains(date.Date))
            return false;
        return true;
    #region Dutch holidays
    private static DateTime[] dutchHolidays =
        new DateTime(2017,  1,  1), // New Year's Day/Nieuwjaarsdag
        new DateTime(2017,  4, 14), // Good Friday/Goede Vrijdag
        new DateTime(2017,  4, 17), // Easter Monday/ Pasen
        new DateTime(2017,  4, 27), // King's Day/Koningsdag
        new DateTime(2017,  5,  5), // Liberation Day/Bevrijdingsdag
        new DateTime(2017,  5, 25), // Ascension Day/Hemelvaartsdag
        new DateTime(2017,  6,  5), // Whit Monday/Pinksteren
        new DateTime(2017, 12, 25), // Christmas Day/Eerste kerstdag
        new DateTime(2017, 12, 26), // St. Stephen's Day/Tweede kerstdag
        new DateTime(2018,  1,  1), // New Year's Day/Nieuwjaarsdag
        new DateTime(2018,  3, 30), // Good Friday/Goede Vrijdag
        // Lots and lots of dates...
        new DateTime(9999,  5,  6), // Ascension Day/Hemelvaartsdag
        new DateTime(9999,  5, 17), // Whit Monday/Pinksteren
        new DateTime(9999, 12, 25), // Christmas Day/Eerste kerstdag
        new DateTime(9999, 12, 26), // St. Stephen's Day/Tweede kerstdag

My old computer isn't happy about having to compile 71,918 lines of C# in a single file, but it's doable, and as far as I can tell, Visual Studio caches the result of compilation, so as long as I don't change the file, there's little adverse effect.

Unit tests #

In order to verify that the implementation works, I wrote this parametrised test:

public class DateTimeExtensionsTests
    [InlineData("2017-03-06""2017-03-06")] // Monday
    [InlineData("2017-03-07""2017-03-07")] // Tuesday
    [InlineData("2017-03-08""2017-03-08")] // Wednesday
    [InlineData("2017-03-09""2017-03-09")] // Thursday
    [InlineData("2017-03-10""2017-03-10")] // Friday
    [InlineData("2017-03-11""2017-03-10")] // Saturday
    [InlineData("2017-03-12""2017-03-10")] // Sunday
    [InlineData("2017-04-14""2017-04-13")] // Good Friday
    [InlineData("2017-04-15""2017-04-13")] // Saturday after Good Friday
    [InlineData("2017-04-16""2017-04-13")] // Sunday after Good Friday
    [InlineData("2017-04-17""2017-04-13")] // Easter Monday
    [InlineData("2017-05-25""2017-05-24")] // Ascension Day - Thursday
    [InlineData("2110-05-26""2110-05-23")] // Whit Monday
    [InlineData("9713-05-05""9713-05-04")] // Liberation Day
    public void AdjustToLatestPrecedingDutchBankDayReturnsCorrectResult(
        string sutS,
        string expectedS)
        var sut = DateTimeOffset.Parse(sutS);
        var actual = sut.AdjustToLatestPrecedingDutchBankDay();
        Assert.Equal(DateTimeOffset.Parse(expectedS), actual);

All test cases pass. This works in [Theory], but unfortunately, it turns out, it doesn't work in practice.

When used in an ASP.NET Web API application, AdjustToLatestPrecedingDutchBankDay throws a StackOverflowException. It took me a while to figure out why, but it turns out that the stack size is smaller in IIS than when you run a 'normal' .NET process, such as an automated test.

System.DateTime is a value type, and as far as I can tell, it uses some stack space during initialisation. When the DateTimeExtensions class is first used, the static dutchHolidays array is initialised, and that uses enough stack space to exhaust the stack when running in IIS.

Final implementation #

The stack space problem seems to be related to DateTime initialisation. If I store a similar number of 64-bit integers in an array, it seems that there's no problem.

First, I had to modify the formatHoliday function:

let formatHoliday (h : Model.PublicHoliday) =
    let t, y, m, d = h.Date.Ticks, h.Date.Year, h.Date.Month, h.Date.Day
    sprintf "%19iL, // %i-%02i-%02i%s/%s" t y m d h.Name h.LocalName

This enabled me to generate a new file with C# code fragments, but now containing ticks instead of DateTime values. Copying those C# fragments into my file gave me this:

public static class DateTimeExtensions
    public static DateTimeOffset AdjustToLatestPrecedingDutchBankDay(
        this DateTimeOffset value)
        var candidate = value;
        while (!(IsDutchBankDay(candidate.DateTime)))
            candidate = candidate.AddDays(-1);
        return candidate;
    private static bool IsDutchBankDay(DateTime date)
        if (date.DayOfWeek == DayOfWeek.Saturday)
            return false;
        if (date.DayOfWeek == DayOfWeek.Sunday)
            return false;
        if (dutchHolidays.Contains(date.Date.Ticks))
            return false;
        return true;
    #region Dutch holidays
    private static long[] dutchHolidays =
         636188256000000000L, // 2017-01-01, New Year's Day/Nieuwjaarsdag
         636277248000000000L, // 2017-04-14, Good Friday/Goede Vrijdag
         636279840000000000L, // 2017-04-17, Easter Monday/ Pasen
         636288480000000000L, // 2017-04-27, King's Day/Koningsdag
         636295392000000000L, // 2017-05-05, Liberation Day/Bevrijdingsdag
         636312672000000000L, // 2017-05-25, Ascension Day/Hemelvaartsdag
         636322176000000000L, // 2017-06-05, Whit Monday/Pinksteren
         636497568000000000L, // 2017-12-25, Christmas Day/Eerste kerstdag
         636498432000000000L, // 2017-12-26, St. Stephen's Day/Tweede kerstdag
         636503616000000000L, // 2018-01-01, New Year's Day/Nieuwjaarsdag
         636579648000000000L, // 2018-03-30, Good Friday/Goede Vrijdag
        // Lots and lots of dates...
        3155171616000000000L, // 9999-05-06, Ascension Day/Hemelvaartsdag
        3155181120000000000L, // 9999-05-17, Whit Monday/Pinksteren
        3155372928000000000L, // 9999-12-25, Christmas Day/Eerste kerstdag
        3155373792000000000L, // 9999-12-26, St. Stephen's Day/Tweede kerstdag

That implementation still passes all tests, and works at in practice as well.

Conclusion #

It took me some time to find a satisfactory solution. I had more than once false start, until I ultimately arrived at the solution I've described here. I consider it simple because it's self-contained, deterministic, easy to understand, and fairly easy to maintain. I even left a comment in the code (not shown here) that described how to recreate the configuration data using the F# script shown here.

The first solution that comes into your mind may not be the simplest solution, but if you take some time to consider alternatives, you may save yourself and your colleagues some future grief.


What do you think about creating an "admin page" that would allow users to configure the bank holidays themselves (which would then be persisted in the application database)? This moves the burden of correctness to the administrators of the application, who I'm sure are highly motivated to get this right - as well as maintain it. It also removes the need for a deployment in the face of changing holidays.

For the sake of convenience, you could still "seed" the database with the values generated by your F# script

2017-04-25 20:00 UTC

jadarnel27, thank you for writing. Your suggestion could be an option as well, but is hardly the simplest solution. In order to implement that, I'd need to add an administration web site for my application, program the user interface, connect the administration site and my original application (a REST API) to a persistent data source, write code for input validation, etcetera.

Apart from all that work, the bank holidays would have to be stored in an out-of-process data store, such as a database or NoSQL data store, because the REST API that needs this feature is running in a server farm. This adds latency to each lookup, as well as a potential error source. What should happen if the connection to the data store is broken? Additionally, such a data store should be backed up, so we'd also need to establish an operational procedure to ensure that that happens.

It was never a requirement that the owners of the application should be able to administer this themselves. It's certainly an option, but it's so much more complex than the solution outlined above that I think one should start by making a return-on-investment analysis.

2017-04-26 7:55 UTC

Another option: Calculate the holidays! I think you might find som useful code in my HolidaysAPI. It is even in F#, and the Web project is based upon a course or blog post by you.

2017-04-26 19:26 UTC

Alf, thank you for writing. Apart from the alternative library, how is that different from the option I covered under the heading Option: Nager.Date?

2017-04-27 5:38 UTC

Great post, thanks. The minor point here is that it is probably not so effective to do Contains over long[]. I'd consider using something that can check the value existance faster.

2017-04-30 19:00 UTC

EQR, thank you for writing. The performance profile of the implementation wasn't my main concern with this article, so it's likely that it can be improved.

I did do some lackadaisical performance testing, but didn't detect any measurable difference between the implementation shown here, and one using a HashSet. On the other hand, there are other options I didn't try at all. One of these could be to perform a binary search, since the array is already ordered.

2017-05-15 11:03 UTC

Hi Mark, thanks for this post. One small note on your selection of bank holidays. Liberation day is an official bank holiday, but only for civil servants, for us 'normal' people this only happens every 5 years. This is reflected in Nager.Date wrong here

2017-07-25 13:35 UTC

Thomas, thank you for writing. That just goes to show, I think, that holiday calculation is as complicated as any other 'business logic'. It should be treated as such, and not as an algorithmic calculation.

2017-07-25 15:12 UTC

A reusable ApiController Adapter

Thursday, 30 March 2017 10:17:00 UTC

Refactor ApiController Adapters to a single, reusable base class.

Regular readers of this blog will know that I write many RESTful APIs in F#, using ASP.NET Web API. Since I like to write functional F#, but ASP.NET Web API is an object-oriented framework, I prefer to escape the object-oriented framework as soon as possible. (In general, it makes good architectural sense to write most of your code as framework-independent as possible.)

(Regular readers will also have seen the above paragraph before.)

To bridge the gap between the object-oriented framework and my functional code, I implement Controller classes like this:

type CreditCardController (imp) =
    inherit ApiController ()
    member this.Post (portalId : string, req : PaymentDtr) : IHttpActionResult =
        match imp portalId req with
        | Success (resp : PaymentDtr-> this.Ok resp :> _
        | Failure RouteFailure -> this.NotFound () :> _
        | Failure (ValidationFailure msg) -> this.BadRequest msg :> _
        | Failure (IntegrationFailure msg) ->
            this.InternalServerError (InvalidOperationException msg) :> _

The above example is a Controller that handles incoming payment data. It immediately delegates all work to an injected imp function, and pattern matches on the return value in order to return correct responses.

This CreditCardController extends ApiController in order to work nicely with ASP.NET Web API, while the injected imp function is written using functional programming. If you want to be charitable, you could say that the Controller is an Adapter between ASP.NET Web API and the functional F# API that actually implements the service. If you want to be cynical, you could also call it an anti-corruption layer.

This works well, but tends to become repetitive:

open System
open System.Web.Http
type BoundaryFailure =
| RouteFailureValidationFailure of stringIntegrationFailure of string
type HomeController (imp) =
    inherit ApiController ()
    member this.Get () : IHttpActionResult =
        match imp () with
        | Success (resp : HomeDtr-> this.Ok resp :> _
        | Failure RouteFailure -> this.NotFound () :> _
        | Failure (ValidationFailure msg) -> this.BadRequest msg :> _
        | Failure (IntegrationFailure msg) ->
            this.InternalServerError (InvalidOperationException msg) :> _
type CreditCardController (imp) =
    inherit ApiController ()
    member this.Post (portalId : string, req : PaymentDtr) : IHttpActionResult =
        match imp portalId req with
        | Success (resp : PaymentDtr-> this.Ok resp :> _
        | Failure RouteFailure -> this.NotFound () :> _
        | Failure (ValidationFailure msg) -> this.BadRequest msg :> _
        | Failure (IntegrationFailure msg) ->
            this.InternalServerError (InvalidOperationException msg) :> _
type CreditCardRecurrentStartController (imp) =
    inherit ApiController ()
    member this.Post (portalId : string, req : PaymentDtr) : IHttpActionResult =
        match imp portalId req with
        | Success (resp : PaymentDtr-> this.Ok resp :> _
        | Failure RouteFailure -> this.NotFound () :> _
        | Failure (ValidationFailure msg) -> this.BadRequest msg :> _
        | Failure (IntegrationFailure msg) ->
            this.InternalServerError (InvalidOperationException msg) :> _
type CreditCardRecurrentController (imp) =
    inherit ApiController ()
    member this.Post (portalId : string, transactionKey : string, req : PaymentDtr) : 
            IHttpActionResult =
        match imp portalId transactionKey req with
        | Success (resp : PaymentDtr-> this.Ok resp :> _
        | Failure RouteFailure -> this.NotFound () :> _
        | Failure (ValidationFailure msg) -> this.BadRequest msg :> _
        | Failure (IntegrationFailure msg) ->
            this.InternalServerError (InvalidOperationException msg) :> _
type PushController (imp) =
    inherit ApiController ()
    member this.Post (portalId : string, req : PushRequestDtr) : IHttpActionResult =
        match imp portalId req with
        | Success () -> this.Ok () :> _
        | Failure RouteFailure -> this.NotFound () :> _
        | Failure (ValidationFailure msg) -> this.BadRequest msg :> _
        | Failure (IntegrationFailure msg) ->
            this.InternalServerError (InvalidOperationException msg) :> _

At this point in our code base, we had five Controllers, and they all had similar implementations. They all pass their input parameters to their injected imp functions, and they all pattern match in exactly the same way. There are, however, small variations. Notice that one of the Controllers expose an HTTP GET operation, whereas the other four expose a POST operation. Conceivably, there could also be Controllers that allow both GET and POST, and so on, but this isn't the case here.

The parameter list for each of the action methods also vary. Some take two arguments, one takes three, and the GET method takes none.

Another variation is that HomeController.Get is annotated with an [<AllowAnonymous>] attribute, while the other action methods aren't.

Despite these variations, it's possible to make the code less repetitive.

You'd think you could easily refactor the code by turning the identical pattern matches into a reusable function. Unfortunately, it's not so simple, because the methods used to return HTTP responses are all protected (to use a C# term for something that F# doesn't even have). You can call this.Ok () or this.NotFound () from a derived class, but not from a 'free' let-bound function.

After much trial and error, I finally arrived at this reusable base class:

type private Imp<'inp, 'out> = 'inp -> Result<'out, BoundaryFailure>
type FunctionController () =
    inherit ApiController ()
    // Imp<'a,'b> -> 'a -> IHttpActionResult
    member this.Execute imp req : IHttpActionResult =
        match imp req with
        | Success resp -> this.Ok resp :> _
        | Failure RouteFailure -> this.NotFound () :> _
        | Failure (ValidationFailure msg) -> this.BadRequest msg :> _
        | Failure (IntegrationFailure msg) ->
            this.InternalServerError (InvalidOperationException msg) :> _

It's been more than a decade, I think, since I last used inheritance to enable reuse, but in this case I could find no other way because of the design of ApiController. It gets the job done, though:

type HomeController (imp : Imp<_, HomeDtr>) =
    inherit FunctionController ()
    member this.Get () = this.Execute imp ()
type CreditCardController (imp : Imp<_, PaymentDtr>) =
    inherit FunctionController ()
    member this.Post (portalId : string, req : PaymentDtr) =
        this.Execute imp (portalId, req)
type CreditCardRecurrentStartController (imp : Imp<_, PaymentDtr>) =
    inherit FunctionController ()
    member this.Post (portalId : string, req : PaymentDtr) =
        this.Execute imp (portalId, req)
type CreditCardRecurrentController (imp : Imp<_, PaymentDtr>) =
    inherit FunctionController ()
    member this.Post (portalId : string, transactionKey : string, req : PaymentDtr) =
        this.Execute imp (portalId, transactionKey, req)
type PushController (imp : Imp<_, unit>) =
    inherit FunctionController ()
    member this.Post (portalId : string, req : PushRequestDtr) =
        this.Execute imp (portalId, req)

Notice that all five Controllers now derive from FunctionController instead of ApiController. The HomeController.Get method is still annotated with the [<AllowAnonymous>] attribute, and it's still a GET operation, whereas all the other methods still implement POST operations. You'll also notice that the various Post methods have retained their varying number of parameters.

In order to make this work, I had to make one trivial change: previously, all the imp functions were curried, but that doesn't fit into a single reusable base implementation. If you consider the type of FunctionController.Execute, you can see that the imp function is expected to take a single input value of the type 'a (and return a value of the type Result<'b, BoundaryFailure>). Since any imp function can only take a single input value, I had to uncurry them all. You can see that now all the Post methods pass their input parameters as a single tuple to their injected imp function.

You may be wondering about the Imp<'inp, 'out> type alias. It's not strictly necessary, but helps keep the code clear. As I've attempted to indicate with the code comment above the Execute method, it's generic. When used in a derived Controller, the compiler can infer the type of 'a because it already knows the types of the input parameters. For example, in CreditCardController.Post, the input parameters are already annotated as string and PaymentDtr, so the compiler can easily infer the type of (portalId, req).

On the other hand, the compiler can't infer the type of 'b, because that value doesn't relate to IHttpActionResult. To help the compiler, I introduced the Imp type, because it enabled me to concisely annotate the type of the output value, while using a wild-card type for the input type.

I wouldn't mind getting rid of Controllers altogether, but this is as close as I've been able to get with ASP.NET Web API.


I find a few issues with your blog post. Technically, I don't think anything you write in it is wrong, but there are a few of the statements I find problematic in the sense that people will probably read the blog post and conclude that "this is the only solution because of X" when in fact X is trivial to circumvent.

The main issue I have with your post is the following:

You'd think you could easily refactor the code by turning the identical pattern matches into a reusable function. Unfortunately, it's not so simple, because the methods used to return HTTP responses are all protected (to use a C# term for something that F# doesn't even have). You can call this.Ok () or this.NotFound () from a derived class, but not from a 'free' let-bound function.

As stated earlier, there is nothing in your statement that is technically wrong, but I'd argue that you've reached the wrong conclusion due to lack of data. Or in this case familiarity with Web API and it's source code. If you go look at the source for ApiController, you will notice that while true, the methods are protected, they are also trivial one-liners (helper methods) calling public APIs. It is completely possible to create a helper ala:

// NOTE: Untested
let handle (ctrl: ApiController) = function
  | Success of result -> OkNegotiatedContentResult (result, ctrl)
  | Failure RouteFailure -> NotFoundResult (ctrl)
  | Failure (ValidationFailure msg) -> BadRequestErrorMessageResult (msg, ctrl)
  | Failure (IntegrationFailure msg) -> ExceptionResult ((InvalidOperationException msg), ctrl)

Another thing you can do is implement IHttpActionResult on your discriminate union, so you can just return it directly from you controller, in which case you'd end up with code like this:

member this.Post (portalId : string, req : PaymentDtr) = imp portalId req

It's definitely a bit more work, but in no way is it undoable. Thirdly, Web API (and especially the new MVC Core which has taken over for it) is incredibly pluggable through it's DI. You don't have to use the base ApiController class if you don't want to. You can override the resolution logic, the handing of return types, routing, you name it. It should for instance be entirely doable to write a handler for "controllers" that look like this:

module MyController =
  // assume impl is imported and in scope here - this controller has no DI
  let post (portalId : string) (req : PaymentDtr) = impl portalId req

This however would probably be a bit of work requiring quite a bit of reflection voodoo. But somebody could definitely do it and put it in a library, and it would be nuget-installable for all.

Anyways, I hope this shows you that there are more options to solve the problem. And who knows, you may have considered all of them and concluded them unviable. I've personally dug through the source code of both MVC and Web API a few times which is why I know a bunch of this stuff, and I just figured you might want to know some of it too :).

2017-03-30 22:06 UTC

Aleksander, thank you for writing. I clearly have some scars from working with ASP.NET Web API since version 0.6 (or some value in that neighbourhood). In general, if a method is defined on ApiController, I've learned the hard way that such methods tend to be tightly coupled to the Controller. Often, such methods even look into the untyped context dictionary in the request message object, so I've gotten used to use them whenever they're present.

These scars have prevented me from pushing the envelope on the 'new' methods that return various IHttpActionResult objects, but as you write, they truly are thin wrappers over constructors.

This does, indeed, enable us to write the mapping of result values as a let-bound function. Thank you for pointing that out!

Mine ended up looking like the following. We've added a few more success and error cases since I originally wrote this article, so it looks more complex than the initial example.

let toHttpResult (controller : ApiController) result : IHttpActionResult = 
    match result with
    | Success (OK resp) -> OkNegotiatedContentResult (resp, controller) :> _
    | Success (Created (resp, location)) ->
        CreatedNegotiatedContentResult (location, resp, controller) :> _
    | Failure RouteFailure -> NotFoundResult controller :> _
    | Failure (ValidationFailure msg) ->
        BadRequestErrorMessageResult (msg, controller) :> _
    | Failure (IntegrationFailure msg) ->
        let resp =
            controller.Request.CreateErrorResponse (
        ResponseMessageResult resp :> _
    | Failure StabilityFailure ->
        let resp =
            new HttpResponseMessage (HttpStatusCode.ServiceUnavailable)
        resp.Headers.RetryAfter <-
            RetryConditionHeaderValue (TimeSpan.FromMinutes 5.)
        ResponseMessageResult resp :> _

Letting my BoundaryFailure discriminated union implement IHttpActionResult sounds promising, but how would that be possible?

The interface has a single method with the type CancellationToken -> Task<HttpResponseMessage>. In order to create e.g. NotFoundResult, you need either an HttpRequestMessage or an ApiController, and none of those are available when IHttpActionResult.ExecuteAsync is invoked.

When it comes to not having to derive from ApiController, I'm aware that even on the full .NET framework (we're not on .NET Core), 'all' the framework needs are IHttpController instances. As you imply, plugging into the framework is likely to be non-trivial, and so far, I haven't found that such an effort would be warranted. I might use it if someone else wrote it, though :)

2017-03-31 13:13 UTC

Dependency rejection

Thursday, 02 February 2017 08:56:00 UTC

In functional programming, the notion of dependencies must be rejected. Instead, applications should be composed from pure and impure functions.

This is the third article in a small article series called from dependency injection to dependency rejection. In the previous article in the series, you learned that dependency injection can't be functional, because it makes everything impure. In this article, you'll see what to do instead.

Indirect input and output #

One of the first concepts you learned when you learned to program was that units of operation (functions, methods, procedures) take input and produce output. Input is in the form of input parameters, and output is in the form of return values. (Sometimes, though, a method returns nothing, but we know from category theory that nothing is also a value (called unit).)

In addition to such input and output, a unit with dependencies also take indirect input, and produce indirect output:

A unit with dependencies and direct and indirect input and output.

When a unit queries a dependency for data, the data returned from the dependency is indirect input. In the restaurant reservation example used in this article series, when tryAccept calls readReservations, the returned reservations are indirect input.

Likewise, when a unit invokes a dependency, all arguments passed to that dependency constitute indirect output. In the example, when tryAccept calls createReservation, the reservation value it uses as input argument to that function call becomes output. The intent, in this case, is to save the reservation in a database.

From indirect output to direct output #

Instead of producing indirect output, you can refactor functions to produce direct output.

A unit with dependencies and direct input and output, but no indirect output.

Such a refactoring is often problematic in mainstream object-oriented languages like C# and Java, because you wish to control the circumstances in which the indirect output must be produced. Indirect output often implies side-effects, but perhaps the side-effect must only happen when certain conditions are fulfilled. In the restaurant reservation example, the desired side-effect is to add a reservation to a database, but this must only happen when the restaurant has sufficient remaining capacity to serve the requested number of people. Since languages like C# and Java are statement-based, it can be difficult to separate the decision from the action.

In expression-based languages like F# and Haskell, it's trivial to decouple decisions from effects.

In the previous article, you saw a version of tryAccept with this signature:

// int -> (DateTimeOffset -> Reservation list) -> (Reservation -> int) -> Reservation
// -> int option

The second function argument, with the type Reservation -> int, produces indirect output. The Reservation value is the output. The function even violates Command Query Separation and returns the database ID of the added reservation, so that's additional indirect input. The overall function returns int option: the database ID if the reservation was added, and None if it wasn't.

Refactoring the indirect output to direct output is easy, then: just remove the createReservation function and return the Reservation value instead:

// int -> (DateTimeOffset -> Reservation list) -> Reservation -> Reservation option
let tryAccept capacity readReservations reservation =
    let reservedSeats =
        readReservations reservation.Date |> List.sumBy (fun x -> x.Quantity)
    if reservedSeats + reservation.Quantity <= capacity
    then { reservation with IsAccepted = true } |> Some
    else None

Notice that this refactored version of tryAccept returns a Reservation option value. The implication is that the reservation was accepted if the return value is a Some case, and rejected if the value is None. The decision is embedded in the value, but decoupled from the side-effect of writing to the database.

This function clearly never writes to the database, so at the boundary of your application, you'll have to connect the decision to the effect. To keep the example consistent with the previous article, you can do this in a tryAcceptComposition function, like this:

// Reservation -> int option
let tryAcceptComposition reservation =
    |> tryAccept 10 (DB.readReservations connectionString)
    |> (DB.createReservation connectionString)

Notice that the type of tryAcceptComposition remains Reservation -> int option. This is a true refactoring. The overall API remains the same, as does the behaviour. The reservation is added to the database only if there's sufficient remaining capacity, and in that case, the ID of the reservation is returned.

From indirect input to direct input #

Just as you can refactor from indirect output to direct output can you refactor from indirect input to direct input.

A unit with dependencies and direct input and output.

Again, in statement-based languages like C# and Java, this may be problematic, because you may wish to defer a query, or base it on a decision inside the unit. In expression-based languages you can decouple decisions from effects, and deferred execution can always be done by lazy evaluation, if that's required. In the case of the current example, however, the refactoring is easy:

// int -> Reservation list -> Reservation -> Reservation option
let tryAccept capacity reservations reservation =
    let reservedSeats = reservations |> List.sumBy (fun x -> x.Quantity)
    if reservedSeats + reservation.Quantity <= capacity
    then { reservation with IsAccepted = true } |> Some
    else None

Instead of calling a (potentially impure) function, this version of tryAccept takes a list of existing reservations as input. It still sums over all the quantities, and the rest of the code is the same as before.

Obviously, the list of existing reservations must come from somewhere, like a database, so tryAcceptComposition will still have to take care of that:

// ('a -> 'b -> 'c) -> 'b -> 'a -> 'c
let flip f x y = f y x
// Reservation -> int option
let tryAcceptComposition reservation =
    |> DB.readReservations connectionString
    |> flip (tryAccept 10) reservation
    |> (DB.createReservation connectionString)

The type and behaviour of this composition is still the same as before, but the data flow is different. First, the function queries the database, which is an impure operation. Then, it pipes the resulting list of reservations to tryAccept, which is now a pure function. It returns a Reservation option that's finally mapped to another impure operation, which writes the reservation to the database if the reservation was accepted.

You'll notice that I also added a flip function in order to make the composition more concise, but I could also have used a lambda expression when invoking tryAccept. The flip function is a part of Haskell's standard library, but isn't in F#'s core library. It's not crucial to the example, though.

Evaluation #

Did you notice that in the previous diagram, above, all arrows between the unit and its dependencies were gone? This means that the unit no longer has any dependencies:

A unit with direct input and output, but no dependencies.

Dependencies are, by their nature, impure, and since pure functions can't call impure functions, functional programming must reject the notion of dependencies. Pure functions can't depend on impure functions.

Instead, pure functions must take direct input and produce direct output, and the impure boundary of an application must compose impure and pure functions together in order to achieve the desired behaviour.

In the previous article, you saw how Haskell can be used to evaluate whether or not an implementation is functional. You can port the above F# code to Haskell to verify that this is the case.

tryAccept :: Int -> [Reservation-> Reservation -> Maybe Reservation
tryAccept capacity reservations reservation =
  let reservedSeats = sum $ map quantity reservations
  in  if reservedSeats + quantity reservation <= capacity
      then Just $ reservation { isAccepted = True }
      else Nothing

This version of tryAccept is pure, and compiles, but as you learned in the previous article, that's not the crucial question. The question is whether the composition compiles?

tryAcceptComposition :: Reservation -> IO (Maybe Int)
tryAcceptComposition reservation = runMaybeT $
  liftIO (DB.readReservations connectionString $ date reservation)
  >>= MaybeT . return . flip (tryAccept 10) reservation
  >>= liftIO . DB.createReservation connectionString

This version of tryAcceptComposition compiles, and works as desired. The code exhibits a common pattern for Haskell: First, gather data from impure sources. Second, pass pure data to pure functions. Third, take the pure output from the pure functions, and do something impure with it.

It's like a sandwich, with the best parts in the middle, and some necessary stuff surrounding it.

Summary #

Dependencies are, by nature, impure. They're either non-deterministic, have side-effects, or both. Pure functions can't call impure functions (because that would make them impure as well), so pure functions can't have dependencies. Functional programming must reject the notion of dependencies.

Obviously, software is only useful with impure behaviour, so instead of injecting dependencies, functional programs must be composed in impure contexts. Impure functions can call pure functions, so at the boundary, an application must gather impure data, and use it to call pure functions. This automatically leads to the ports and adapters architecture.

This style of programming is surprisingly often possible, but it's not a universal solution; other alternatives exist.


Hi, Thank you for this blog post series. I also read your other posts on ports and adapters and the proposed architecture makes sense in terms of how it works, but I struggle to see the benefit in a real world application. Maybe let me explain my question with a quick example.

In the 2nd blog post of this series you demonstrated this function:

// int -> (DateTimeOffset -> Reservation list) -> (Reservation -> int) -> Reservation
// -> int option
let tryAccept capacity readReservations createReservation reservation =
    let reservedSeats =
        readReservations reservation.Date |> List.sumBy (fun x -> x.Quantity)
    if reservedSeats + reservation.Quantity <= capacity
    then createReservation { reservation with IsAccepted = true } |> Some
    else None

If I understand it correctly this function is pure if readReservations and createReservation are both pure otherwise it is impure.

I also understand the benefit of having a pure function, because it is a lot easier to understand the code, test the code and reason about it. That makes sense as well :).

So in the 3rd blog post you make tryAccept a pure function, by removing the function dependencies and replacing it with simple values:

// int -> Reservation list -> Reservation -> Reservation option
let tryAccept capacity reservations reservation =
    let reservedSeats = reservations |> List.sumBy (fun x -> x.Quantity)
    if reservedSeats + reservation.Quantity <= capacity
    then { reservation with IsAccepted = true } |> Some
    else None

However this was only possible because you essentially moved the impure code into another new function:

// Reservation -> int option
let tryAcceptComposition reservation =
    |> DB.readReservations connectionString
    |> flip (tryAccept 10) reservation
    |> (DB.createReservation connectionString)

So after all the application hasn't really reduced the total number of impure functions (still 3 in each case - readReservations, createReservation and tryAccept[Composition]).

The only difference I see is that one impure function has been refactored into 2 functions - one pure and one impure. Considering that the original tryAccept function was already fully testable from a unit testing point of view and quite readable what is the benefit of this additional step? I would almost argue that the original tryAccept function was even easier to read/understand than the combination of tryAccept and tryAcceptComposition. I understand that impure functions like this are not truly functional, but in a real world application you must have some impure functions and I would like to better understand where trade-off benefit of that additional step is? Am I missing something else?

2017-02-03 10:34 UTC

Dustin, thank you for writing. There are several answers to your question, depending on the perspective one is interested in. I'll see if I can cover the most important ones.

Is it functional? #

On the most fundamental level, I'm interested in learning functional programming. In order to do this, I seek out strictly functional solutions to problems. Haskell is a great help in that endeavour, because it's not a hybrid language. It only allows you to do functional programming.

Does it make sense to back-port Haskell solutions to F#, then? That depends on what one is trying to accomplish, but if the goal is nothing but learning how to do it functionally, then that goal is accomplished.

Toy examples #

On another level, the example I've presented here is obviously nothing but a toy example. It's simplified, because if I presented readers with a more realistic example, the complexity of the real problem could easily drown out the message of the example. Additionally, most readers would probably give up reading.

I'm asking my readers to pretend that the problem is more complex than the one I present here; pretend that this problem is a stand-in for a harder problem.

In this particular context, there could be all sorts of complications:

  • Reservations could be for time slots instead of whole dates. In order to keep the example simple, I treat each reservation as simply blocking out an entire date. I once dined at a restaurant where they started serving at 19:00, and if you weren't there on time, you'd miss the first courses. Most restaurants, though, allow you to make reservations for a particular time, and many have more than one serving on a single evening.
  • Most restaurants have tables, not seats. Again, the same restaurant I mentioned above seated 12 people at a bar-like arrangement facing the kitchen, but most restaurants have tables of varying sizes. If they get a reservation for three people, they may have to reserve a table for four.
  • Perhaps the restaurant would like to implement a feature where, if it receives a reservation that doesn't fill out a table (like a reservation for three people, and only four-people tables are left), it'd defer the decision to see if a 'better' reservation arrives later.
  • Some people make reservations, but never show up. For that reason, a restaurant may want to allow a degree of overbooking, just like airlines. How much overbooking to allow is a business decision.
  • A further wrinkle on the overbooking business rule is that you may have a different overbooking policy for Fridays than for, say, Wednesdays.
  • Perhaps the restaurant would like to implement a waiting-list feature as well.
As you can see, we could easily imagine that the business logic could be more convoluted. Keeping all of that decision logic pure would be beneficial.

Separation of concerns #

In my experience, there's an entire category of software defects that occur because of state mutation in business logic. You could have an area of your code that calls other code, which calls other code, and so on, for several levels of nesting. Somewhere, deep in the bowels of such a system, a conditional statement flips a boolean flag that consequently impact how the rest of the program runs. I've seen plenty of examples of such software, and it's inhumane; it doesn't fit within human cognitive limits.

Code that allows arbitrary side-effects is difficult to reason about.

Knowing that an subgraph of your call tree is pure reduces defects like that. This is nothing but another way to restate the command-query separation principle. In F#, we still can't be sure unless we exert some discipline, but in Haskell, all it takes is a look at the type of a function or value. If it doesn't include IO, you know that it's pure.

Separating pure code from impure code is separation of concern. Business logic is one concern, and I/O is another concern, and the better you can separate these, the fewer sources of defects you'll have. True, I haven't reduced the amount of code by much, but I've separated concerns by separating the code that contains (side) effects from the pure code.

Testability #

It's true that the partial application version of tryAccept is testable, because it has isolation, but the tests are more complicated than they have to be:

[<Property(QuietOnSuccess = true)>]
let ``tryAccept behaves correctly when it can accept``
    (NonNegativeInt excessCapacity)
    (expected : int) =
    Tuple2.curry id
    <!> Gen.reservation
    <*> Gen.listOf Gen.reservation
    |>  Arb.fromGen |> Prop.forAll <| fun (reservation, reservations) ->
    let capacity =
        + (reservations |> List.sumBy (fun x -> x.Quantity))
        + reservation.Quantity
    let readReservations = ((=!) reservation.Date) >>! reservations
    let createReservation =
        ((=!) { reservation with IsAccepted = true }) >>! expected
    let actual =
        tryAccept capacity readReservations createReservation reservation
    Some expected =! actual
[<Property(QuietOnSuccess = true)>]
let ``tryAccept behaves correctly when it can't accept``
    (PositiveInt lackingCapacity) =
    Tuple2.curry id
    <!> Gen.reservation
    <*> Gen.listOf Gen.reservation
    |>  Arb.fromGen |> Prop.forAll <| fun (reservation, reservations) ->
    let capacity =
        (reservations |> List.sumBy (fun x -> x.Quantity)) - lackingCapacity
    let readReservations _ = reservations
    let createReservation _ = failwith "Mock shouldn't be called."
    let actual =
        tryAccept capacity readReservations createReservation reservation
    None =! actual

(You can find these tests in commit d2387cceb81eabc349a63ab7df1249236e9b1d13 in the accompanying sample code repository.) Contrast those dependency-injection style tests to these tests against the pure version of tryAccept:

[<Property(QuietOnSuccess = true)>]
let ``tryAccept behaves correctly when it can accept``
    (NonNegativeInt excessCapacity) =
    Tuple2.curry id
    <!> Gen.reservation
    <*> Gen.listOf Gen.reservation
    |>  Arb.fromGen |> Prop.forAll <| fun (reservation, reservations) ->
    let capacity =
        + (reservations |> List.sumBy (fun x -> x.Quantity))
        + reservation.Quantity
    let actual = tryAccept capacity reservations reservation
    Some { reservation with IsAccepted = true } =! actual
[<Property(QuietOnSuccess = true)>]
let ``tryAccept behaves correctly when it can't accept``
    (PositiveInt lackingCapacity) =
    Tuple2.curry id
    <!> Gen.reservation
    <*> Gen.listOf Gen.reservation
    |>  Arb.fromGen |> Prop.forAll <| fun (reservation, reservations) ->
    let capacity =
        (reservations |> List.sumBy (fun x -> x.Quantity)) - lackingCapacity
    let actual = tryAccept capacity reservations reservation
    None =! actual

They're simpler, and since they don't use mocks, they're more robust. They were easier to write, and I subscribe to the spirit of GOOS: if test are difficult to write, the system under test should be simplified.

2017-02-05 20:09 UTC

Hi Mark,

Thanks for your talk at NDC last month, and for writing this series! I feel that the functional community (myself included) has a habit of using examples that aren't obviously relevant to the sort of line-of-business programming most of us do in our day jobs, so articles like this are sorely needed.

We talked a little about this in person after your talk at the conference: I wanted to highlight a potential criticism of this style of programming. Namely, there's still some important business logic being carried out by your tryAcceptComposition function, like checking the capacity on the requested reservation date. How do you unit test that readReservations is called with the correct date? Likewise, how do you unit test that rejected reservations don't get saved? Real world business logic isn't always purely functional in nature. Sometimes the side effects that your code performs are part of the requirements.

The Haskell philosophy isn't about rejecting side effects outright - it's about measuring and controlling them. I wouldn't write tryAcceptComposition using IO. Instead I'd program to the interface, not the implementation, using an mtl-style class to abstract over monads which support saving and loading reservations.

class Monad m => MonadReservation m where
    readReservations :: ConnectionString -> Date -> m [Reservation]
    createReservation :: ConnectionString -> Reservation -> m ReservationId

tryAcceptComposition :: MonadReservation m => Reservation -> m (Maybe ReservationId)
tryAcceptComposition r = runMaybeT $ do
    reservations <- lift $ readReservations connectionString (date r)
    accepted <- MaybeT $ return $ tryAccept 10 reservations r
    lift $ createReservation connectionString accepted

Code that lives in a MonadReservation context can read and create reservations in the database but nothing else; it doesn't have all the power of IO. During unit testing I can use an instance of MonadReservation that returns canned values, and in production I can use a monad that actually talks to the database.

Since type classes are syntactic sugar for passing an argument, this is really just a nicer way of writing your original DI-style code. I don't advocate the "free monad" style that's presently trendy in Scala-land because I find it unnecessarily complex. 90% of the purported advantages of free monads are already supported by simpler language features.

I suppose the main downside of this design is that you can't express it in F#, at least not cleanly. It relies on type classes and higher-kinded types.

Hope you find this interesting, I'd love to hear what you think!


2017-02-06 16:28 UTC

Benjamin, thank you for writing. The alternative you propose looks useful in Haskell, but, as you've already suggested, it doesn't translate well into F#.

I write F# code professionally, whereas so far, I've only used Haskell to critique my F# code. (If someone who reads this comment would offer to pay me to write some Haskell code, please get in touch.) In other words, I still have much to learn about Haskell. I think I understand as much, however, that I'd be able to use your suggested design to unit test tryAcceptComposition using the Identity monad for Stubs, or perhaps MonadWriter or MonadState for Mocks. I'll have to try that one day...

In F#, I write integration tests. Such tests are important regardless, and often they more closely relate to actual requirements, so I find this a worthwhile effort anyway.

2017-02-11 22:42 UTC

Hi Mark,

thanks for the post series, which I find interesting and needed. There is one part of your post that I find deserves further exploration. You write:

in statement-based languages like C# and Java, this may be problematic, because you may wish to defer a query, or base it on a decision inside the unit. In expression-based languages you can decouple decisions from effects, and deferred execution can always be done by lazy evaluation, if that's required.
Firstly, I would say that you can write expression-based programs in any language that has expressions, which naturally includes C# and Java. But that's not particularly relevant to this discussion.

More to the point, you're glossing over this as though it were a minor detail, when in fact I don't think it is. Let's explore the case in which "you may wish to defer a query, or base it on a decision inside the unit". The way you do this "by lazy evaluation" would be - I assume - by passing a function as an argument to your unit. But this is then effectively dependency injection, because you're passing in a function which has side effects, which will be called (or not) from the unit.

So, it seems to me that your technique of extracting side effects out of the unit provides a good general guideline, but not a completely general way to replace dependency injection.

2017-02-16 11:47 UTC

Enrico, thank you for writing. There's a lot to unpack in that quote, which was one of the reasons I didn't expand it. It would have made the article too long, and wandered off compared to its main point. I don't mind going into these details here, though.

Direction of data #

In order to get the obvious out of the way first, the issue you point out is with my refactoring of indirect input to direct input. Refactoring from indirect to output to direct output is, as far as I can tell, not on your agenda. Designing with direct input in mind seems uncontroversial to me, so that makes sense.

No hard rules #

On this blog, I often write articles as I figure out how to deal with problems. Sometimes, I report on my discoveries at a time where I've yet to accumulate years of experience. What I've learned so far is that dependency injection isn't functional. What I'm still exploring is what to do instead.

It's my experience that the type of refactoring I demonstrate here can surprisingly often be performed. I don't want to claim that it's always possible to do it like this. In fact, I'm still looking for good examples where this will not be possible. Whenever I think of a simple enough example that I could share it here, I always realise that if only I simplify the problem, I can put it into the shape seen here.

My thinking is, however, constrained by my professional experience. I've been doing web (service) development for so many years now that it constraints my imagination. When you execution scope is exclusively a single HTTP request at a time, you tend to keep things simple. I'd welcome a simplified, but still concrete example where the impure/pure/impure sandwich described here isn't going to be possible.

This may seem like a digression, but my point is that I don't claim to be the holder of a single, undeniable truth. Still, I find that this article describes a broadly applicable design and implementation technique.

Language specifics #

The next topic we need to consider is our choice of language. When I wrote that deferred execution can always be done by lazy evaluation, that's exactly how Haskell works. Haskell is lazily evaluated, so any value passed as direct input can be unevaluated until required. That goes for IO as well, but then, as we've learned, you can't pass impure data to a pure function.

All execution is, in that sense, deferred, unless explicitly forced. Thus, any potential need for deferred execution has no design implications.

F#, on the other hand, is an eagerly evaluated language, so there, deferred execution may have design implications.

Performance #

Perhaps it's my lack of imagination again, but I can't think of a well-designed system where deferred execution is required for purposes of correctness. As far as I can tell, deferred execution is a performance concern. You wish to defer execution of a query because that operation takes significant time.

That's a real concern, but I often find that people worry too much about performance. Again, this is probably my lack of wider experience, as I realise that performance can be important in smart phone apps, games, and the like. Clearly, performance is also important in the world of REST APIs, but I've met a lot of people who worry about performance without ever measuring it.

When you start measuring performance, you'll often be surprised to discover where your code spends actual time. So my design approach is always to prioritise making the system work first, and then, if there are performance problems, figure out how to tweak it so that it becomes satisfactory. In my experience, such tweaking is only necessary now and then. I'm not claiming that my code is the fastest it could be, but it's often fast enough, and as easy to maintain as I can make it.

The need for data #

Another concern is the need for data. If you consider the above tryAccept function, it always uses reservations. Thus, there's no gain in deferring the database query, because you'll always need the data.

Deferred execution is only required in those cases where you have conditional branching, and only in certain cases do you need to read a particular piece of data.

Even conditional branching isn't enough of a criterion, though, because you could have branching where, in 99.9 % of the cases, you'd be performing the query anyway. Would you, then, need deferred execution for the remaining 0.1 % of the cases?

Lazy sequences #

Still, let's assume that we've implemented a system using pure functions that take pure data, but to our dismay we discover that there's one query that takes time to execute, and that we truly only need it some of the time. In .NET, there are two distinct situations:

  • We need a scalar value
  • We need a collection of values
If we need a collection of values, we only need to make a minuscule change to the design of our function. Instead of taking an F# list, or an array, as direct input, we can make the function take a sequence (IEnumerable<T> in C#) as input. These can be implemented as lazily evaluated sequences, which gives us the deferred execution we need.

Lazy scalar values #

This leaves the corner case where we need a lazily evaluated scalar value. In such cases, I may have to make a concession to performance in my function design, but I wouldn't change the argument to a function, but rather to a lazy value.

Lazy values are deferred, but memoised, which is the reason I'd prefer them over function arguments.

2017-02-18 19:54 UTC

In a previous comment, you said:

I'd welcome a simplified, but still concrete example where the impure/pure/impure sandwich described here isn't going to be possible.

I have on two occasions stumbled into cases where I can't find a good way to pull this off. The reason may be that there's a workflow seemingly consisting of several impure steps interleaved with pure decisions. The cases are similar, so I will share one of them as an example.

We have an API for allowing users to register. We also have a separate API for two-factor authentication (2FA). When users register, they have to complete a "proof" using the 2FA API to verify ownership of their mobile number.

The 2FA API has two relevant endpoints used internally by our other APIs: One is used for creating a proof, and returns a proof ID that can be passed on to the API client. This endpoint is used when a client makes a request without a proof ID, or with an invalid proof ID. The other endpoint is used for verifying that a proof has been completed. This endpoint is used when a client supplies a proof ID in the request.

(The 2FA API also has endpoints the client uses to complete the proof, but that is not relevant here.)

When a user registers, this is the workflow:

A flowchart describing the workflow for completing a registration.

Here is a simple implementation of the workflow using "dependency injection" (I will skip the actual composition function, similar to your tryAcceptComposition, which is not interesting here):

let completeRegistrationWorkflow
    (createProof: Mobile -> Async<ProofId>)
    (verifyProof: Mobile -> ProofId -> Async<bool>)
    (completeRegistration: Registration -> Async<unit>)
    (proofId: ProofId option)
    (registration: Registration)
    : Async<CompleteRegistrationResult> =
  async {
    match proofId with
    | None ->
        let! proofId = createProof registration.Mobile
        return ProofRequired proofId
    | Some proofId ->
        let! isValid = verifyProof registration.Mobile proofId
        if isValid then
          do! completeRegistration registration
          return RegistrationCompleted
          let! proofId = createProof registration.Mobile
          return ProofRequired proofId

There are two decisions that are pure and could conceivably be extracted to a pure (non-DI) function, indicated by blue in the flowchart:

  • Initially: Should we create a new proof or verify an existing proof? (based on whether the client supplied a proof ID)
  • After verifying a supplied proof: Should we complete the registration or create a new proof? (based on whether the supplied proof ID was valid)

My question then, is this: Is it possible to refactor this to direct input/output, in a way that actually reduces complexity where it matters? (In other words, in a way where the decisions mentioned above are part of a pure, non-DI function?)

Otherwise, I just want to say that this series has helped me become a better F# programmer. Since originally reading it, I have consistently tried to design for direct input/output as opposed to using "dependency injection", and my code has become better for it. Thanks!

2019-11-06 10:06 UTC

Christer, thank you for writing. That was such a fruitful question that I wrote a new article in order to answer it. I hope you find it useful, but if not, let's continue the discussion there or here, dependening on what makes most sense.

2019-12-02 13:22 UTC

Hi Mark, excelent post series, thank you. Right after finishing this post I had the same questions Dustin Moris Gorski had and I think I still have some, even after your answer.

For simplicity's sake, I'll represent TryAccept as 3 operations: read, calculate and maybe create. Both read and create are injected so that I can change their implementations at runtime, if necessary. That is the same exact representation for both the C# and F# codes that use DI, and also the tryAcceptComposition code, with the difference that tryAcceptComposition depends on the actual implementation of DB, so it's relatively more brittle than the DI alternatives.

Although TryAccept and TryAcceptComposition are doing the same thing, I'm still trying to think why the latter looks more functional than the first and whether DI is really not necessary. I got to 2 conclusions and I wanted to know your opinion about them:

1) The difference between the 2 implementations (TryAccept and TryAcceptComposition) is that the first is following an imperative style, while the second is purely declaritive, with one big composition. Both implementations perform the exact same operations, evoking the same dependencies, but with different code styles.

2) If we try and take your style of sandwich to extremes and push dependencies to the very edges of the application, to the point where it's just a simple big composition with dependencies at the beginning and end, we my replace "injection" with import statements, importing our dependencies in the same place the composition is written. I don't think this would work in every scenario as operations in the middle of this composition would need access to dependencies too (which could them be pushed out like TryAccept, but could make the code less readable). Do you think this is doable?

2020-01-14 20:37 UTC

Danilo, thank you for writing. Regarding your first point, I'm not sure I follow. Both functions are written in F#, which is an expression-based language (at least when you ignore the interop parts). Imperative code is usually defined as coding with assignments rather than expressions. There's no assignments here.

Also, tryAccept and tryAcceptComposition aren't equivalent. One is a specialisation of the other, so to speak. You can't change the behaviour of tryAcceptComposition unless you edit the actual code. You can, on the other hand, change the observable behaviour of tryAccept by composing it with different impure actions.

As to your second observation:

"I don't think this would work in every scenario as operations in the middle of this composition would need access to dependencies too"
I'm still keen on getting some compelling examples of this. Christer van der Meeren kindly tried to supply such an example, but it turned out as another fine sandwich. This happens conspicuously often.

I've never claimed that the sandwich architecture is always possible, but every time I try to find a compelling counter-example, it usually turns out to be refactorable into a sandwich after all. Do, however, refer to the comments to that later article.

2020-01-19 21:07 UTC

Hi Mark, I found myself thinking about this early this morning and wondering about scenarios where injecting dependencies may still be "functional" (i.e. pure) and would like your input. It also incorporates TDD (or at least testing strategies).

Here's my example: I was recently refactoring some code that was structured like this (C#-ish pseudo-code):

				if (shouldNotify(record, supportingRecord, settings)) 
					record.status = calculateStatusAfterEmail(record, supportingRecord);
					record.status = calculateStatusWithoutEmail(record, supportingRecord);	
Let's gloss over the side-effects all over the place here and just focus on the flow.

To ease unit testing, I extracted these private methods into a separate interface and am injecting it into this flow. Call me lazy, but I wanted to just mock/stub the results of these pure calls when testing the overall flow and wrote separate tests on the implementation of the new interface that could focus on all the different combinations of values. Is that a strategy you would support/recommend, or would you hesitate to extract pure functions from an implementation just to ease testing? I also convinced myself that I could justify this on the grounds of following the Single Responsibility Principle (i.e. this flow should not need to be modified if the status calculation logic changes).

Your thoughts? By the way, your new dependency injection strategies is on its way to me, perhaps that book will provide me with your views on that, but I couldn't resist posting it here.

2020-05-28 14:30 UTC

Sven, thank you for writing. I think that your question deserves a longer answer than what you'll get here. It's a topic that for some time I've been contemplating giving a more detailed treatment. It's a rich enough subject that a couple of articles would be required. I'm still pondering how to organise such content.

The short answer is that I'm increasingly looking for alternatives to the sort of interaction-based testing you imply. I think that my article series on the benefits of state-based testing outlines most of my current thinking on the topic.

2020-06-03 5:46 UTC

I read over all the comments and I would like clarification on when logic is required for resolving the dependency being provided. Benjamin Hodgson asked this:

Namely, there's still some important business logic being carried out by your tryAcceptComposition function, like checking the capacity on the requested reservation date. How do you unit test that readReservations is called with the correct date?

Suppose the readRervations didn't just get all reservations from the dabatase just to process for one restaurant or on one day. Thus the restaurant ID and date may be needed to fetch the correct reservations, and this logic would have to happen outside of the unit. I think you touched on this complication in a response above, but I'd like to confirm if I am understanding your responses correctly: you simply perform this logic outside of the unit and have an integration test for the entire workflow?

2021-12-15 22:11 UTC

Chris, thank you for writing. There are more nuances to that question than might be immediately apparent. At least, I'll answer the question in several ways.

First, I'd like to challenge the implicit assumption that one must always test everything. What to test and what not to test depend on multiple independent forces, such as how costly an error might be, how likely an error is to occur, and so on.

Take, as an example, the code base that accompanies my book Code That Fits in Your Head. This code base is also a restaurant reservation system, but has a realistic level of complexity, so I think it might better highlight the sort of considerations that might be warranted. The code that is most relevant to the question is this snippet from ReservationsController.TryCreate:

var reservations = await Repository
    .ReadReservations(restaurant.Id, reservation.At)
var now = Clock.GetCurrentDateTime();
if (!restaurant.MaitreD.WillAccept(now, reservations, reservation))
    return NoTables500InternalServerError();
await Repository.Create(restaurant.Id, reservation)

How likely is it that the code changes in such a way that it calls ReadReservations with the wrong input? Either because the original programmer who wrote the code made a mistake, or because a later developer introduced a change in which this code calls ReadReservations with incorrect input?

Now, such errors certainly do occur. Programmers are human, and to err is human.

Still, it's important to keep in mind that the risk of introducing an error into a code base is proportional to the amount of code. The more code, the more errors you'll have. This goes for test code as well. You can also, inadvertently, introduce errors into a test.

You can counteract some of that by writing the test first. That's the reasons it's so important to see a test fail. Only by seeing it fail can you feel confident that it verifies something real.

So, depending on circumstances, it may be relevant and important to write a test that verifies that ReadReservations is called with the correct arguments. Still, I think it's important to weigh advantages and disadvantages before blindly insisting that it's necessary to protect against such a contingency.

When I originally wrote the above code, I wrote no unit test that explicitly verify whether ReadReservations is called with the correct arguments. After all, in that context, there's only one DateTime value in scope: reservation.At. (This is not entirely true, because now is also a DateTime value, but as the code is written, it's not available for ReadReservations because it's retrieved after the ReadReservations call. But see below.)

In a context like this, you'd have to go out of your way to call ReadReservations with the wrong date. You could create a value on the spot, like this:

var reservations = await Repository
    .ReadReservations(restaurant.Id, new DateTime(2021, 12, 19, 18, 30, 0))

or this:

var reservations = await Repository
    .ReadReservations(restaurant.Id, reservation.At.AddDays(1))

None of these are impossible, but I do consider them to be conspicuous. You don't easily make those kinds of mistakes. You'd almost have to go out of your way to make these mistakes, and a good pairing partner or code review ought to catch such a mistake.

Another option is if someone finds a good reason to reorder the code:

var now = Clock.GetCurrentDateTime();
var reservations = await Repository
    .ReadReservations(restaurant.Id, now)

In this example, now is available to ReadReservations, and perhaps a later edit might introduce the error of calling it with now.

This is perhaps a more realistic, honest mistake.

None of the 171 unit tests in the code base catch any of the above three mistakes.

Is the code base unsafe, then?

No, because as luck would have it, an integration test (NoOverbookingRace) fails on any of these three edits. I admit that this is just a fortunate accident. I added that particular integration test for another reason: to reproduce an unrelated bug that occurred in 'production'.

I usually try to base the amount of testing I ask from a team on the quality of the team. The more I trust that they pair-program or perform regular code reviews, the less I insist on test coverage, and vice versa.

In summary, though, if I find that it's important to verify that ReadReservations was called with the correct arguments, I'd favour a state-based integration test.

2021-12-19 12:24 UTC

Partial application is dependency injection

Monday, 30 January 2017 12:40:00 UTC

The equivalent of dependency injection in F# is partial function application, but it isn't functional.

This is the second article in a small article series called from dependency injection to dependency rejection.

People often ask me how to do dependency injection in F#. That's only natural, since I wrote Dependency Injection in .NET some years ago, and also since I've increasingly focused my energy on F# and other functional programming languages.

Over the years, I've seen other F# experts respond to that question, and often, the answer is that partial function application is the F# way to do dependency injection. For some years, I believed that as well. It turns out to be true in one sense, but incorrect in another. Partial application is equivalent to dependency injection. It's just not a functional solution to dealing with dependencies.

(To be as clear as I can be: I'm not claiming that partial application isn't functional. What I claim is that partial application used for dependency injection isn't functional.)

Attempted dependency injection using functions #

Returning to the example from the previous article, you could try to rewrite MaîtreD.TryAccept as a function:

// int -> (DateTimeOffset -> Reservation list) -> (Reservation -> int) -> Reservation
// -> int option
let tryAccept capacity readReservations createReservation reservation =
    let reservedSeats =
        readReservations reservation.Date |> List.sumBy (fun x -> x.Quantity)
    if reservedSeats + reservation.Quantity <= capacity
    then createReservation { reservation with IsAccepted = true } |> Some
    else None

You could imagine that this tryAccept function is part of a module called MaîtreD, just to keep the examples as equivalent as possible.

The function takes four arguments. The first is the capacity of the restaurant in question; a primitive integer. The next two arguments, readReservations and createReservation fill the role of the injected IReservationsRepository in the previous article. In the object-oriented example, the TryAccept method used two methods on the repository: ReadReservations and Create. Instead of using an interface, in the F# function, I make the function take two independent functions. They have (almost) the same types as their C# counterparts.

The first three arguments correspond to the injected dependencies in the previous MaîtreD class. The fourth argument is a Reservation value, which corresponds to the input to the previous TryAccept method.

Instead of returning a nullable integer, this F# version returns an int option.

The implementation is also equivalent to the C# example: Read the relevant reservations from the database using the readReservations function argument, and sum over their quantities. Based on the number of already reserved seats, decide whether or not to accept the reservation. If you can accept the reservation, set IsAccepted to true, call the createReservation function argument, and pipe the returned ID (integer) to Some. If you can't accept the reservation, then return None.

Notice that the first three arguments are 'dependencies', whereas the last argument is the 'actual input', if you will. This means that you can use partial function application to compose this function.

Application #

If you recall the definition of the previous IMaîtreD interface, the TryAccept method was defined like this (C# code snippet):

int? TryAccept(Reservation reservation);

You could attempt to define a similar function with the type Reservation -> int option. Normally, you'd want to do this closer to the boundary of the application, but the following example demonstrates how to 'inject' real database operations into the function.

Imagine that you have a DB module with these functions:

module DB =
    // string -> DateTimeOffset -> Reservation list
    let readReservations connectionString date = // ..
    // string -> Reservation -> int
    let createReservation connectionString reservation = // ..

The readReservations function takes a connection string and a date as arguments, and returns a list of reservations for that date. The createReservation function also takes a connection string, as well as a reservation. When invoked, it creates a new record for the reservation and returns the ID of the newly created row. (This sort of API violates CQS, so you should consider alternatives.)

If you partially apply these functions with a valid connection string, both have the type desired for their roles in tryAccept. This means that you can create a function from these elements:

// Reservation -> int option
let tryAcceptComposition =
    let read   = DB.readReservations  connectionString
    let create = DB.createReservation connectionString
    tryAccept 10 read create

Notice how tryAccept itself is partially applied. Only the arguments corresponding to the C# dependencies are passed to it, so the return value is a function that 'waits' for the last argument: the reservation. As I've attempted to indicate by the code comment above the function, it has the desired type of Reservation -> int option.

Equivalence #

Partial application used like this is equivalent to dependency injection. To see how, consider the generated Intermediate Language (IL).

F# is a .NET language, so it compiles to IL. You can decompile that IL to C# to get a sense of what's going on. If you do that with the above tryAcceptComposition, you get something like this:

internal class tryAcceptComposition@17 : FSharpFunc<ReservationFSharpOption<int>>
    public int capacity;
    public FSharpFunc<Reservationint> createReservation;
    public FSharpFunc<DateTimeOffsetFSharpList<Reservation>> readReservations;
    internal tryAcceptComposition@17(
        int capacity,
        FSharpFunc<DateTimeOffsetFSharpList<Reservation>> readReservations,
        FSharpFunc<Reservationint> createReservation)
        this.capacity = capacity;
        this.readReservations = readReservations;
        this.createReservation = createReservation;
    public override FSharpOption<int> Invoke(Reservation reservation)
        return MaîtreD.tryAccept<int>(
            this.capacity, this.readReservations, this.createReservation, reservation);

I've cleaned it up a bit, mostly by removing all attributes from the various elements. Notice how this is a class, with class fields, and a constructor that takes values for the fields and assigns them. It's constructor injection!

Partial application is dependency injection.

It compiles, works as expected, but is it functional?

Evaluation #

People sometimes ask me: How do I know whether my F# code is functional?

I sometimes wonder about that myself, but unfortunately, as nice a language as F# is, it doesn't offer much help in that regard. Its emphasis is on functional programming, but it allows mutation, object-oriented programming, and even procedural programming. It's a friendly and forgiving language. (This also makes it a great 'beginner' functional language, because you can learn functional concepts piecemeal.)

Haskell, on the other hand, is a strictly functional language. In Haskell, you can only write your code in the functional way.

Fortunately, F# and Haskell are similar enough that it's easy to port F# code to Haskell, as long as the F# code already is 'sufficiently functional'. In order to evaluate if my F# code is properly functional, I sometimes port it to Haskell. If I can get it to compile and run in Haskell, I take that as confirmation that my code is functional.

I've previously shown an example similar to this one, but I'll repeat the experiment here. Will porting tryAccept and tryAcceptComposition to Haskell work?

It's easy to port tryAccept:

tryAccept :: Int -> (ZonedTime -> [Reservation]) -> (Reservation -> Int) -> Reservation
             -> Maybe Int
tryAccept capacity readReservations createReservation reservation =
  let reservedSeats = sum $ map quantity $ readReservations $ date reservation
  in  if reservedSeats + quantity reservation <= capacity
      then Just $ createReservation $ reservation { isAccepted = True }
      else Nothing

Clearly, there are differences, but I'm sure that you can also see the similarities. The most important feature of this function is that it's pure. All Haskell functions are pure by default, unless explicitly declared to be impure, and that's not the case here. This function is pure, and so are both readReservations and createReservation.

The Haskell version of tryAccept compiles, but what about tryAcceptComposition?

Like the F# code, the experiment is to see if it's possible to 'inject' functions that actually operate against a database. Equivalent to the F# example, imagine that you have this DB module:

readReservations :: ConnectionString -> ZonedTime -> IO [Reservation]
readReservations connectionString date = -- ..

createReservation :: ConnectionString -> Reservation -> IO Int
createReservation connectionString reservation = -- ..

Database operations are, by definition, impure, and Haskell admirably models that with the type system. Notice how both functions return IO values.

If you partially apply both functions with a valid connection string, the IO context remains. The type of DB.readReservations connectionString is ZonedTime -> IO [Reservation], and the type of DB.createReservation connectionString is Reservation -> IO Int. You can try to pass them to tryAccept, but the types don't match:

tryAcceptComposition :: Reservation -> IO (Maybe Int)
tryAcceptComposition reservation =
  let read   = DB.readReservations  connectionString
      create = DB.createReservation connectionString
  in tryAccept 10 read create reservation

This doesn't compile.

It doesn't compile, because the database operations are impure, and tryAccept wants pure functions.

In short, partial application used for dependency injection isn't functional.

Summary #

Partial application in F# can be used to achieve a result equivalent to dependency injection. It compiles and works as expected, but it's not functional. The reason it's not functional is that (most) dependencies are, by their very nature, impure. They're either non-deterministic, have side-effects, or both, and that's often the underlying reason that they are factored into dependencies in the first place.

Pure functions, however, can't call impure functions. If they could, they would become impure themselves. This rule is enforced by Haskell, but not by F#.

When you inject impure operations into an F# function, that function becomes impure as well. Dependency injection makes everything impure, which explains why it isn't functional.

Functional programming solves the problem of decoupling (side) effects from program logic another way. That's the topic of the next article.

Next: Dependency rejection.


Kurren Nischal
A very useful read as always.
A couple of questions: If you're porting your code from F# to Haskell and back into F#, why not just use Haskell in the first place?
Also, why would you want to mark a function as having side effects in the first place?
2017-07-17 14:10 UTC

Kurren, thank you for writing. Why not use Haskell in the first place? In some situations, if that's an option, I'd go for it. In most of my professional work, however, it's not. A bit of background is required, I think. I've spent most of my professional career working with Microsoft technologies. Most of my clients use .NET. The majority of my clients use C#, but some are interested in adopting functional programming. F# is a great bridge to functional programming, because it integrates so well with existing C# code.

Perhaps most importantly is that while these organisations learn F# as a new programming language, and a new paradigm, all other dimensions of software development remain the same. You can use familiar libraries and frameworks, you can use the same Continuous Integration and build tools as you've been used to, you can deploy like you always do, and you can operate and monitor your software like before.

If you start using Haskell, not only will developers have to learn an entire new ecosystem, but if you have a separate operations team, they would have be on board with that as well.

If you can't get the entire organisation to accept Haskell-based software, then F# is a great choice for a .NET shop. Developers will obviously know the difference, but the rest of the organisation can be oblivious to the choice of language.

When it comes to side-effects, that's one of the main motivations behind pure code in the first place. A major source of bugs is that often, programs have unintended side-effects. It's easy to introduce defects in a code base when side-effects are uncontrolled. That's the case for languages like C# and Java, and, unfortunately, F# as well. When you look at a method or function, there's no easy way to determine whether it has side-effects. In other words: all methods or functions could have side effects. (Also: all methods could return null.)

This slows you down when you work on an existing code base, because there's only one way to determine if side-effects or non-deterministic behaviour are part of the code you're about to call: you have to read it all.

On the other hand, if you can tell, just by looking at a function's type, whether or not it has side-effects, you can save yourself a lot of time. By definition, all the pure functions have no side-effect. You don't need to read the code in order to detect that.

In Haskell, this is guaranteed. In F#, you can design your system that way, but some discipline is required in order to uphold such a guarantee.

2017-07-18 09:31 UTC

Dependency injection is passing an argument

Friday, 27 January 2017 09:27:00 UTC

Is dependency injection really just passing an argument? A brief review.

This is the first article in a small article series called from dependency injection to dependency rejection.

In a talk at the 2012 Northeast Scala Symposium, Rúnar Bjarnason casually remarked that dependency injection is "really just a pretentious way to say 'taking an argument'". Given that I've written a 500+ pages book about dependency injection, you might expect me to disagree with that. Yet, there's some truth to that statement, although it's not quite as simple as that.

In this article, I'll show you some simple examples and explain why, on the one hand, Rúnar Bjarnason is right, but also, on the other hand, why there's a bit more to it.

Restaurant reservation example #

Like the other articles in this series, the example scenario is on-line restaurant reservation. Imagine that you've been asked to develop an HTTP-based API that accepts JSON documents containing restaurant reservations. Furthermore, assume that you're using ASP.NET Web API with C# for the job, and that you're aspiring to use domain-driven design.

In order to handle the incoming POST request, you could write an action method like this:

public IHttpActionResult Post(ReservationRequestDto dto)
    var validationMsg = validator.Validate(dto);
    if (validationMsg != "")
        return this.BadRequest(validationMsg);
    var r = mapper.Map(dto);
    var id = maîtreD.TryAccept(r);
    if (id == null)
        return this.StatusCode(HttpStatusCode.Forbidden);
    return this.Ok();

This method follows a simple and familiar path: validate input, map to a domain model, delegate to said model, examine posterior state, and return a result.

You may have noticed, though, that this method doesn't do all the work itself. It delegates some of the work to collaborators: validator, mapper, and maîtreD. Where do these collaborators come from?

They are dependencies. Could you make the Post method take them as arguments?

Unfortunately, you can't. The Post method constitutes part of the boundary of the HTTP API. ASP NET Web API routes and dispatches incoming HTTP requests by convention, and action methods must follow that convention. You can't just make the function take any argument you'd like, so you have to find another place to pass those dependencies to the object.

The second-best option (after the Post method itself) is via the constructor:

public ReservationsController(
    IValidator validator,
    IMapper mapper,
    IMaîtreD maîtreD)
    this.validator = validator;
    this.mapper = mapper;
    this.maîtreD = maîtreD;

This is the application of a design pattern called constructor injection. It captures the dependencies in class fields, making them available for members (like Post) of the class.

This turns out to be a regular pattern.

Turtles all the way down #

You could argue that the Post method is a special case, since it's part of the boundary of the system, and therefore must adhere to specific rules. On the other hand, these rule don't apply deeper in the implementation, so could you implement other objects by simply passing in dependencies as arguments?

Consider, as an example, the implementation of IMaîtreD.TryAccept:

public int? TryAccept(Reservation reservation)
    var reservedSeats = reservationsRepository
        .Sum(r => r.Quantity);
    if (reservedSeats + reservation.Quantity <= capacity)
        reservation.IsAccepted = true;
        return reservationsRepository.Create(reservation);
    return null;

This method has another collaborator: reservationsRepository. It's another dependency. Where does it come from?

Could you make the TryAccept method take reservationsRepository as an argument?

Unfortunately, that's not possible either, because the method is defined by the IMaîtreD interface:

public interface IMaîtreD
    int? TryAccept(Reservation reservation);

You may recall that the above Post method is programmed against the IMaîtreD interface, and not the concrete class. It'd be a leaky abstraction to add IReservationsRepository as an argument to IMaîtreD.TryAccept, because not all implementations of the interface may need that dependency. Or perhaps another implementation has another dependency. Should we add that to the parameter list of IMaîtreD.TryAcceptas well?

Surely, that's not a tenable design principle. On the other hand, by using constructor injection, you can decouple implementation details from your abstractions:

public MaîtreD(int capacity, IReservationsRepository reservationsRepository)
    this.capacity = capacity;
    this.reservationsRepository = reservationsRepository;

This constructor not only takes an IReservationsRepository object, but also an integer that represents the capacity of the restaurant in question. This demonstrates that dependencies can also be primitive values.

Summary #

Dependency injection is, in a sense, only a specific way for objects to take arguments. Often, however, objects have roles defined by the interfaces they implement. Such objects may need collaborators that are not available via the APIs defined by these interfaces, so you'll have to supply dependencies via members that belong to the concrete class in question. Passing dependencies via a class' constructor is the best way to do that.

Next: Partial application is dependency injection.


Hey Mark. You may want to update the article slightly to cover the use of the [FromServices] attribute, which allows you to inject any service into a controller action as method-level injection. The main article on it is this one.

I'm personally not a fan of using it, but it is an option that transforms standard injection into parameter-passing for controllers.

2020-02-23 15:11 UTC

Juliano, thank you for writing. I wasn't aware of that particular capability, so thank you for bringing it to my attention.

My article attempts to explain a general software design problem, as well as a possible solution. While it uses ASP.NET as an example context, the main point is independent of any particular framework, or language, for that matter.

2021-02-24 14:09 UTC

From dependency injection to dependency rejection

Friday, 27 January 2017 07:55:00 UTC

The problem typically solved by dependency injection in object-oriented programming is solved in a completely different way in functional programming.

Several years ago, I wrote a book called Dependency Injection in .NET, which was published in 2011. The book contains examples in C#, but since then I've increasingly become interested in functional programming to the extend that I now consider F# my primary language.

With that combination, it's no wonder that people often ask me how to do dependency injection in functional programming.

I've seen more than one answer, from other people, explaining how partial function application is equivalent to dependency injection. In a small series of articles, I'll explain both why this is true, but also why it's not functional. I'll conclude by showing a functional alternative to decoupling logic and (side) effects.

Bob: How do I do dependency injection in Scala? Other man: You don't, because Scala is a functional language. Bob: Fine, it's functional. How do I inject dependencies? Other man: You use a free monad which allows you to build a monad from any Functor. Bob: Did you just tell me to go fuck myself? Other man: I believe I did, Bob.

(Comic courtesy of John Muellerleile and Igal Tabachnik.)

There's another school of functional programmers who believe that dependency injection in functional programming involves a Free monad.

You can often make do with less, though.

In my experience, it's usually enough to refactor a unit to take only direct input and output, and then compose an impure/pure/impure 'sandwich'. You'll see an example later.

This article series contains the following parts:

  1. Dependency injection is passing an argument
  2. Partial application is dependency injection
  3. Dependency rejection
  4. Pure interactions
The first three articles revolve around a common example, which is one of my favourite scenarios: on-line restaurant reservation. You can see an actual example client in my Functional Architecture with F# Pluralsight course. The (somewhat dated) client source code is available on GitHub. The server-side F# and Haskell example code for this article series is available on GitHub.

The scenario is to implement an HTTP-based API that can accept incoming JSON documents that represent restaurant reservations.

The fourth article on pure interactions is a gateway to another article series on free monads.

I should point out that nowhere in this article series do I reject dependency injection as a set of object-oriented patterns. In object-oriented programming, dependency injection is a well-known and comprehensively described way to achieve decoupling and testability. In the next article, you'll see a brief review of dependency injection in C#.

Next: Dependency injection is passing an argument.

Decoupling application errors from domain models

Tuesday, 03 January 2017 12:26:00 UTC

How to prevent application-specific error cases from infecting your domain models.

Functional error-handling is often done with the Either monad. If all is good, the right case is returned, but if things go wrong, you'll want to return a value that indicates the error. In an application, you'll often need to be able to distinguish between different kinds of errors.

From application errors to HTTP responses #

When an application encounters an error, it should respond appropriately. A GUI-based application should inform the user about the error, a batch job should log it, and a REST API should return the appropriate HTTP status code.

Regular readers of this blog will know that I write many RESTful APIs in F#, using ASP.NET Web API. Since I like to write functional F#, but ASP.NET Web API is an object-oriented framework, I prefer to escape the object-oriented framework as soon as possible. (In general, it makes good architectural sense to write most of your code as framework-independent as possible.)

In my Test-Driven Development with F# Pluralsight course (a free, condensed version is also available), I demonstrate how to handle various error cases in a Controller class:

type ReservationsController (imp) =
    inherit ApiController ()
    member this.Post (dtr : ReservationDtr) : IHttpActionResult =
        match imp dtr with
        | Failure (ValidationError msg) -> this.BadRequest msg :> _
        | Failure CapacityExceeded ->
            this.StatusCode HttpStatusCode.Forbidden :> _
        | Success () -> this.Ok () :> _

The injected imp function is a complete, composed, vertical feature implementation that performs both input validation, business logic, and data access. If input validation fails, it'll return Failure (ValidationError msg), and that value is translated to a 400 Bad Request response. Likewise, if the business logic returns Failure CapacityExceeded, the response becomes 403 Forbidden, and a success is returned as 200 OK.

Both ValidationError and CapacityExceeded are cases of an Error type. This is only a simple example, so these are the only cases defined by that type:

type Error =
| ValidationError of stringCapacityExceeded

This seems reasonable, but there's a problem.

Error infection #

In F#, a function can't use a type unless that type is already defined. This is a problem because the Error type defined above mixes different concerns. If you seek to make illegal states unrepresentable, it follows that validation is not a concern in your domain model. Validation is still important at the boundary of an application, so you can't just ignore it. The ValidationError case relates to the application boundary, while CapacityExceeded relates to the domain model.

Still, when implementing your domain model, you may want to return a CapacityExceeded value from time to time:

// int -> int -> Reservation -> Result<Reservation,Error>
let checkCapacity capacity reservedSeats reservation =
    if capacity < reservation.Quantity + reservedSeats
    then Failure CapacityExceeded
    else Success reservation

Notice how the return type of this function is Result<Reservation,Error>. In order to be able to implement your domain model, you've now pulled in the Error type, which also defines the ValidationError case. Your domain model is now polluted by an application boundary concern.

I think many developers would consider this trivial, but in my experience, failure to manage dependencies is the dominant reason for code rot. It makes the code less general, and less reusable, because it's now coupled to something that may not fit into a different context.

Particularly, the situation in the example looks like this:

Dependency diagram

Boundary and data access modules depend on the domain model, as they should, but everything depends on the Error type. This is wrong. Modules or libraries should be able to define their own error types.

The Error type belongs in the Composition Root, but it's impossible to put it there because F# prevents circular dependencies (a treasured language feature).

Fortunately, the fix is straightforward.

Mapped Either values #

A domain model should be self-contained. As Robert C. Martin puts it in APPP:

Abstractions should not depend upon details. Details should depend upon abstractions.
Your domain model is an abstraction of the real world (that's why it's called a model), and is the reason you're developing a piece of software in the first place. So start with the domain model:

type BookingError = CapacityExceeded
// int -> int -> Reservation -> Result<Reservation,BookingError>
let checkCapacity capacity reservedSeats reservation =
    if capacity < reservation.Quantity + reservedSeats
    then Failure CapacityExceeded
    else Success reservation

In this example, there's only a single type of domain error (CapacityExceeded), but that's mostly because this is an example. Real production code could define a domain error union with several cases. The crux of the matter is that BookingError isn't infected with irrelevant implementation details like validation error types.

You're still going to need an exhaustive discriminated union to model all possible error cases for your particular application, but that type belongs in the Composition Root. Accordingly, you also need a way to return validation errors in your validation module. Often, a string is all you need:

// ReservationDtr -> Result<Reservation,string>
let validateReservation (dtr : ReservationDtr) =
    match dtr.Date |> DateTimeOffset.TryParse with
    | (true, date) -> Success {
        Reservation.Date = date
        Name = dtr.Name
        Email = dtr.Email
        Quantity = dtr.Quantity }
    | _ -> Failure "Invalid date."

The validateReservation function returns a Reservation value when validation succeeds, and a simple string with an error message if it fails.

You could, conceivably, return string values for errors from many different places in your code, so you're going to map them into an appropriate error case that makes sense in your application.

In this particular example, the Controller shown above should still look like this:

type Error =
| ValidationError of stringDomainError
type ReservationsController (imp) =
    inherit ApiController ()
    member this.Post (dtr : ReservationDtr) : IHttpActionResult =
        match imp dtr with
        | Failure (ValidationError msg) -> this.BadRequest msg :> _
        | Failure DomainError -> this.StatusCode HttpStatusCode.Forbidden :> _
        | Success () -> this.Ok () :> _

Notice how similar this is to the initial example. The important difference, however, is that Error is defined in the same module that also implements ReservationsController. This is part of the composition of the specific application.

In order to make that work, you're going to need to map from one failure type to another. This is trivial to do with an extra function belonging to your Result (or Either) module:

// ('a -> 'b) -> Result<'c,'a> -> Result<'c,'b>
let mapFailure f x =
    match x with
    | Success succ -> Success succ
    | Failure fail -> Failure (f fail)

This function takes any Result value and maps the failure case instead of the success case. It enables you to transform e.g. a BookingError into a DomainError:

let imp candidate = either {
    let! r = validateReservation candidate |> mapFailure ValidationError
    let  i = SqlGateway.getReservedSeats connectionString r.Date
    let! r = checkCapacity 10 i r |> mapFailure (fun _ -> DomainError)
    return SqlGateway.saveReservation connectionString r }

This composition is a variation of the composition I've previously published. The only difference is that the error cases are now mapped into the application-specific Error type.

Conclusion #

Errors can occur in diverse places in your code base: when validating input, when making business decisions, when writing to, or reading from, databases, and so on.

When you use the Either monad for error handling, in a strongly typed language like F#, you'll need to define a discriminated union that models all the error cases you care about in the specific application. You can map module-specific error types into such a comprehensive error type using a function like mapFailure. In Haskell, it would be the first function of the Bifunctor typeclass, so this is a well-known function.



Why is it a problem to use HttpStatusCode in the domain model. They appear to be a standard way of categorizing errors.

2017-02-07 13:37 UTC

David, thank you for writing. The answer depends on your goals and definition of domain model.

I usually think of domain models in terms of separation of concerns. The purpose of a domain model is to model the business logic, and as Martin Fowler writes in PoEAA about the Domain Model pattern, "you'll want the minimum of coupling from the Domain Model to other layers in the system. You'll notice that a guiding force of many layering patterns is to keep as few dependencies as possible between the domain model and the other parts of the system."

In other words, you're separating the concern of implementing the business rules from the concerns of being able to save data in a database, render it on a screen, send emails, and so on. While also important, these are separate concerns, and I want to be able to vary those independently.

People often hear statements like that as though I want to reserve myself the right to replace my SQL Server database with Neo4J (more on that later, though!). That's actually not my main goal, but I find that if concerns are mixed, all change becomes harder. It becomes more difficult to change how data is saved in a database, and it becomes harder to change business rules.

The Dependency Inversion Principle tries to address such problems by advising that abstractions shouldn't depend on implementation details, but instead, implementation details should depend on abstractions.

This is where the goals come in. I find Robert C. Martin's definition of software architecture helpful. Paraphrased from memory, he defines a software architect's role as enabling change; not predicting change, but making sure that when change has to happen, it's as economical as possible.

As an architect, one of the heuristics I use is that I try to imagine how easily I can replace one component with another. It's not that I really believe that I may have to replace the SQL Server database with Neo4J, but thinking about how hard it would be gives me some insights about how to structure a software solution.

I also imagine what it'd be like to port an application to another environment. Can I port my web site's business rules to a batch job? Can I port my desktop client to a smart phone app? Again, it's not that I necessarily predict that I'll have to do this, but it tells me something about the degrees of freedom offered by the architecture.

If not explicitly addressed, the opposite of freedom tends to happen. In APPP, Robert C. Martin describes a number of design smells, one of them Immobility: "A design is immobile when it contains parts that could be useful in other systems, but the effort and risk involved with separating those parts from the original system are too great. This is an unfortunate, but very common occurrence."

Almost as side-effect, an immobile system is difficult to test. A unit test is a different environment than the intended environment. Well-architected systems are easy to unit test.

HTTP is a communications protocol. Its purpose is to enable exchange of information over networks. While it does that well, it's specifically concerned with that purpose. This includes HTTP status code.

If you use the heuristic of imagining that you'd have to move the heart of your application to a batch job, status codes like 301 Moved Permanently, 404 Not Found, or 405 Method Not Allowed make little sense.

Using HTTP status codes in a domain model couples the model to a particular environment, at least conceptually. It has little to do with the ubiquitous language that Eric Evans discusses in DDD.

2017-02-07 17:08 UTC

From REST to algebraic data

Friday, 16 December 2016 07:23:00 UTC

Mapping RESTful HTTP requests to values of algebraic data types is easy.

In previous articles, you've seen how to easily model a simple domain model with algebraic data types, and how to use RESTful API design to surface such a model at the boundary of an application. In this article, you'll see how trivial it is to map incoming HTTP requests back to values of algebraic data types.

The advantage of REST is that you can make illegal states unrepresentable. Clients follow links, and while clients are supposed to treat links as opaque values, URLs still contain information your API can use.

Routing and dispatching #

Continuing where the previous article left off, clients can issue POST requests against a URL like On the server, a well-known piece of code handles such requests. (In the example code base I've used so far, I've been using ASP.NET Web API, so the code that handles such a request is a Controller.) Since you know that URLs like that are always routed to that particular piece of code, you can create a new PaymentType value that specifically represents an individual payment with a credit card:

let paymentType = Individual { Name = "credit-card"; Action = "Pay" }

If, on the other hand, the client is using a provided link to POST a representation against the URL, your server-side dispatcher will route the request to a different handler (Controller), in which case you can create a PaymentType value like this:

let paymentType = Parent { Name = "credit-card"; Action = "Pay" }

Finally, if the client has already created a parent payment and is now using the resulting link to create child payments, it may be POSTing to a URL like Your server-side dispatcher will route that request to a third handler. Most web frameworks, including ASP.NET Web API, will be able to pull values out of URLs. In this case, you can configure it so that it pulls the value 42 out of the URL and binds it to a value called transactionKey. With this, again it's trivial to create a PaymentType value:

let paymentType = Child (transactionKey, { Name = "credit-card"; Action = "PayRecurrent" })

Notice that, despite containing different data, and being created three different places in the code base, they all have the same type: PaymentType. This means that you can pass these values to a common pay function, which handles the actual communication with the third-party payment service.

Code reuse #

Independent of the route the data arrived at, a central, reusable function named pay handles all such payments. This is still an impure boundary function that takes various other input apart from PaymentType. Without going into too much detail, it has a type like Config -> PaymentType -> Result<PaymentDtr,BoundaryFailure>. Don't worry if some of the details look obscure; the important point is that pay is a function that takes a PaymentType value as input. You can visualise the transition from HTTP requests to a function call like this:

Three different URLs mapped to three different PaymentType values, which are again passed to the single pay function

The pay function is composed from various smaller functions, some pure and some impure. Ultimately, it transforms all the input data to the format required by the third-party payment service, and forwards the transaction information. Inside that function you'll find the pattern match that you saw in my previous article.

Summary #

By making good use of routing and dispatching, you can easily map incoming HTTP requests to values of algebraic data types. This enables you to close the loop on exposing your domain model at the boundary of your system. Not only can clients request data from your API in terms of your model, but when clients send data to your API, you can translate that data back to your model.

Domain modelling with REST

Wednesday, 07 December 2016 09:15:00 UTC

Make illegal states unrepresentable by using hyperlinks as the engine of application state.

Every piece of software, whether it's a web service, smart phone app, batch job, or speech recognition system, interfaces with the world in some way. Sometimes, that interface is a user interface, sometimes it's a machine-readable interface; sometimes it involves rendering pixels on a screen, and sometimes it involves writing to files, selecting records from a database, sending emails, and so on.

Programmers often struggle with how to model these interactions. This is particularly difficult because at the boundaries, systems no longer adhere to popular programming paradigms. Previously, I've explained why, at the boundaries, applications aren't object-oriented. By the same type of argument, neither are they functional (as in 'functional programming').

If that's the case, why should you even bother with 'domain modelling'? Particularly, does it even matter that, with algebraic data types, you can make illegal states unrepresentable? If you need to compromise once you hit the boundary of your application, is it worth the effort?

It is, if you structure your application correctly. Proper (level 3) REST architecture gives you one way to structure applications in such a way that you can surface the constraints of your domain model to the interface layer. When done correctly, you can also make illegal states unrepresentable at the boundary.

A payment example #

In my previous article, I demonstrated how to use (static) types to model an on-line payment domain. To summarise, my task was to model three types of payments:

  • Individual payments, which happen only once.
  • Parent payments, which start a long-term payment relationship.
  • Child payments, which are automated payments originally authorised by an initial parent payment.
The constraint I had to model is that a child payment requires a transaction key that identifies the original parent payment. When making a payment, however, it's only valid to supply a transaction key for a child payment. It'd be invalid to supply a transaction key to a parent or an individual payment. On the other hand, you have to distinguish individual payments from parent payments, because only parent payments can be used to start a long-term payment relationship. All this leads to this table of possible combinations:
"StartRecurrent" : "false" "StartRecurrent" : "true"
"OriginalTransactionKey" : null Individual Parent
"OriginalTransactionKey" : "1234ABCD" Child (Illegal)
The table shows that it's illegal to simultaneously provide a transaction key and set StartRecurrent to true. The other three combinations, on the other hand, are valid.

As I demonstrated in my previous article, you can easily model this with algebraic data types.

At the boundary, however, there are no static types, so how could you model something like that as a web service?

A RESTful solution #

A major advantage of REST is that it gives you a way to realise your domain model at the application boundary. It does require, though, that you design the API according to level 3 of the Richardson maturity model. In other words, it's not REST if you're merely tunnelling JSON (or XML) through HTTP. It's still not REST if you publish URL templates and expect clients to fill data into specific place-holders of those URLs.

It's REST if the only way a client can interact with your API is by following hyperlinks.

If you follow those design principles, however, it's easy to model the above payment domain as a RESTful API.

In the following, I will show examples in XML, but it could as well have been JSON. After all, a true REST API must support content negotiation. One of the reasons that I prefer XML is that I can use XPath to point out various nodes.

A client must begin at a pre-published 'home' resource, just like the home page of a web site. This resource presents affordances in the shape of hyperlinks. As recommended by the RESTful Web Services Cookbook, I always use ATOM links:

<home xmlns=""
        <atom:link rel="example:pay-individual"
                   href="" />
        <atom:link rel="example:pay-individual"
                   href="" />
        <atom:link rel="example:pay-parent"
                   href="" />

A client receiving the above response is effectively presented with a choice. It can choose to pay with a gift card or credit card, and nothing else, however much it'd like to pay with, say, PayPal. For each of these payment methods, zero or more links are available.

Specifically, there are links to create both an individual or a parent payment with a credit card, but it's only possible to make an individual payment with a gift card. You can't set up a long-term, automated payment relationship with a gift card. (This may or may not make sense, depending on how you look at it, but it's fundamentally a business decision.)

Links are defined by relationship types, which are unique identifiers present in the rel attributes. You can think of them as equivalent to the human-readable text in an HTML a tag that suggests to a human user what to expect from clicking the link; only, rel attribute values are machine-readable and part of the contract between client and service.

Notice how the above XML representation only gives a client the option of making an individual or a parent payment with a credit card. A client can't make a child payment at this point. This follows the domain model, because you can't make a child payment without first having made a parent payment.

RESTful individual payments #

If a client wishes to make an individual payment, it follows the link identified by

/home/payment-methods/payment-method[id = 'credit-card']/links/atom:link[@rel = 'example:pay-individual']/@href

In the above XPath query, I've ignored the default document namespace in order to make the expression more readable. The query returns, and the client can now make an HTTP POST request against that URL with a JSON or XML document containing details about the payment (not shown here, because it's not important for this particular discussion).

RESTful parent payments #

If a client wishes to make a parent payment, the initial procedure is the same. First, it follows the link identified by

/home/payment-methods/payment-method[id = 'credit-card']/links/atom:link[@rel = 'example:pay-parent']/@href

The result of that XPath query is, so the client can make an HTTP POST request against that URL with the payment details. Unlike the response for an individual payment, the response for a parent payment contains another link:

<payment xmlns=""
    <atom:link rel="example:pay-child"
               href="" />
    <atom:link rel="example:payment-details"
               href="" />

This response echoes the details of the payment: this is a payment of 13.37 EUR for invoice 1234567890. It also includes some links that a client can use to further interact with the payment:

  • The example:payment-details link can be used to query the API for details about the payment, for example its status.
  • The example:pay-child link can be used to make a child payment.
The example:pay-child link is only returned if the previous payment was a parent payment. When a client makes an individual payment, this link isn't present in the response, but when the client makes a parent payment, it is.

Another design principle of REST is that cool URIs don't change; once the API has shown a URL like to a client, it should honour that URL indefinitely. The upshot of that is that a client can save that URL for later use. If a client wants to, say, renew a subscription, it can make a new HTTP POST request to that URL a month later, and that's going to be a child payment. Clients don't have to hack the URL in order to figure out what the transaction key is; they can simply store the complete URL as is and use it later.

A network of options #

Using a design like the one sketched above, you can make illegal states unrepresentative. There's no way for a client to make a payment with StartRecurrent = true and a non-null transaction key; there's no link to that combination. Such an API uses hypermedia as the engine of application state.

It shouldn't be surprising that proper RESTful design works that way. After all, REST is essentially a distillate of the properties that make the World Wide Web work. On a human-readable web page, the user follows links to other pages, and a well-designed web site will only enable a link if the destination exists.

You can even draw a graph of the API I've sketched above:

Graph of payment options, including a start node, and end node, and a node for each of the three payment types

In this diagram, you can see that when you make an individual payment, that's all you can do. You can also see that the only way to make a child payment is by first making a parent payment. There's also a path from parent payments directly to the end node, because a client doesn't have to make a child payment just because it made a parent payment.

If you think that this looks like a finite state machine, then that's no coincidence. That's exactly what it is. You have states (the nodes) and paths between them. If a state is illegal, then don't add that node; only add nodes for legal states, then add links between the nodes that model legal transitions.

Incidentally, languages like F# excel at implementing finite state machines, so it's no wonder I like to implement RESTful APIs in F#.

Summary #

Truly RESTful design enables you to make illegal states unrepresentable by using hypermedia as the engine of application state. This gives you a powerful design tool to ensure that clients can only perform correct operations.

As I also wrote in my previous article, this, too, is no silver bullet. You can turn an API into a pit of success, but there are still many fault scenarios that you can't prevent.

If you were intrigued by this article, but are having trouble applying these design techniques to your own field, I'm available for hire for short or long-term engagements.

Page 33 of 68

"Our team wholeheartedly endorses Mark. His expert service provides tremendous value."
Hire me!