Generalised Test Data Builder

Monday, 21 August 2017 06:09:00 UTC

This article presents a generalised Test Data Builder.

This is the second in a series of articles about the relationship between the Test Data Builder design pattern, and the identity functor. The previous article was a review of the Test Data Builder pattern.

Boilerplate

While the Test Data Builder is an incredibly versatile and useful design pattern, it has a problem. In languages like C# and Java, it's difficult to generalise. This leads to an excess of boilerplate code.

Expanding on Nat Pryce's original example, an InvoiceBuilder is composed of other builders:

public class InvoiceBuilder
{
    private Recipient recipient;
    private IReadOnlyCollection<InvoiceLine> lines;
 
    public InvoiceBuilder()
    {
        this.recipient = new RecipientBuilder().Build();
        this.lines = new List<InvoiceLine> { new InvoiceLineBuilder().Build() };
    }
 
    public InvoiceBuilder WithRecipient(Recipient newRecipient)
    {
        this.recipient = newRecipient;
        return this;
    }
 
    public InvoiceBuilder WithInvoiceLines(
        IReadOnlyCollection<InvoiceLine> newLines)
    {
        this.lines = newLines;
        return this;
    }
 
    public Invoice Build()
    {
        return new Invoice(recipient, lines);
    }
}

In order to create a Recipient, a RecipientBuilder is used. Likewise, in order to create a single InvoiceLine, an InvoiceLineBuilder is used. This pattern repeats in the RecipientBuilder:

public class RecipientBuilder
{
    private string name;
    private Address address;
 
    public RecipientBuilder()
    {
        this.name = "";
        this.address = new AddressBuilder().Build();
    }
 
    public RecipientBuilder WithName(string newName)
    {
        this.name = newName;
        return this;
    }
 
    public RecipientBuilder WithAddress(Address newAddress)
    {
        this.address = newAddress;
        return this;
    }
 
    public Recipient Build()
    {
        return new Recipient(this.name, this.address);
    }
}

In order to create an Address object, an AddressBuilder is used.

Generalisation attempts

You can describe the pattern in a completely automatable manner:

  1. For each domain class, create a corresponding Builder class.
  2. For each class field or property in the domain class, define a corresponding field or property in the Builder.
  3. In the Builder's constructor, initialise each field or property with a 'good' default value.
    • If the field is a primitive value, such as a string or integer, hard-code an appropriate value.
    • If the field is a complex domain type, use that type's corresponding Builder to create the default value.
  4. For each class field or property, add a With[...] method that changes the field and returns the Builder itself.
  5. Add a Build method that returns a new instance of the domain class with the constituent values collected so far.
When you can deterministically descrbe an automatable process, you can write code to automate it.

People have already done that. After having written individual Test Data Builders for a couple of months, I got tired of it and wrote AutoFixture. It uses Reflection to build objects at run-time, but I've also witnessed attempts to automate Test Data Builders via automated code generation.

AutoFixture has been moderately successful, but some people find its API difficult to learn. Correspondingly, code generation comes with its own issues.

In languages like C# or Java, it's difficult to identify a better generalisation.

Generic Builder

Instead of trying to automate the Test Data Builder pattern, you can pursue a different strategy. At first, it doesn't look all that promising, but if you soldier on, it'll reveal meaningful insights.

As an alternative to replicating the Test Data Builder pattern exactly, you can define a single generically typed Builder class:

public class Builder<T>
{
    private readonly T item;
 
    public Builder(T item)
    {
        if (item == null)
            throw new ArgumentNullException(nameof(item));
 
        this.item = item;
    }
 
    public Builder<T1> Select<T1>(Func<TT1> f)
    {
        var newItem = f(this.item);
        return new Builder<T1>(newItem);
    }
 
    public T Build()
    {
        return this.item;
    }
 
    public override bool Equals(object obj)
    {
        var other = obj as Builder<T>;
        if (other == null)
            return base.Equals(obj);
 
        return object.Equals(this.item, other.item);
    }
 
    public override int GetHashCode()
    {
        return this.item.GetHashCode();
    }
}

The Builder<T> class reduces the Test Data Builder design patterns to the essentials:

  • A constructor that initialises the Builder with default data.
  • A single fluent interface Select method, which returns a new Builder object.
  • A Build method, which returns the built object.
Perhaps you wonder about the name of the Select method, but there's a good reason for that; you'll learn about it later.

This example of a generic Builder class overrides Equals (and, therefore, also GetHashCode). It doesn't have to do that, but there's a good reason to do this that we'll also come back to later.

It doesn't seem particularly useful, and a first attempt at using it seems to confirm such skepticism:

var address = Build.Address().Select(a =>
{
    a.City = "Paris";
    return a;
}).Build();

This example first uses Build.Address() to create an initial Builder object with appropriate defaults. This static method is defined on the static Build class:

public static Builder<Address> Address()
{
    return new Builder<Address>(new Address("""", PostCode().Build()));
}

Contrary to Builder<T>, which is a reusable, general-purpose class, the static Build class is an example of a collection of Test Utility Methods specific to the domain model you're testing. Notice how the Build.Address() method uses Build.PostCode().Build() to create a default value for the initial Address object's post code.

The above example passes a C# code block to the Select method. It takes the a (Address) object as input, specifically mutates its City property, and returns it. This syntax is crude, but works. It may look acceptable when pinning a single City property, but it quickly becomes awkward:

var invoice = Build.Invoice().Select(i =>
    {
        i.Recipient = Build.Recipient().Select(r =>
        {
            r.Address = Build.Address().WithNoPostCode().Build();
            return r;
        }).Build();
        return i;
    }).Build();

Not only is it difficult to get right when writing such nested statements, it's also hard to read. You can, however, correct that problem, as you'll see in a little while.

Before we commence on making the code prettier, you may have noticed that the Select method returns a Builder with a different generic type argument than it contains. The Select method on a Builder<T> object has the signature public Builder<T1> Select<T1>(Func<T, T1> f). Until now, however, all the examples you've seen return the input object. In those examples, T is the same as T1. For completeness' sake, here's an example of a proper change of type:

var address = Build.PostCode()
    .Select(pc => new Address("Rue Morgue""Paris", pc))
    .Build();

This example uses a Builder<PostCode> to create a new Address object. Plugging in the types, T becomes PostCode, and T1 becomes Address.

Perhaps you noticed that this example looks a little better than the previous examples. Instead of having to supply a C# code block, with return statement and all, this call to Select passes a proper (lambda) expression.

Expressions from extensions

It'd be nice if you could use expressions, instead of full code blocks, with the Select method. As a first step, you could write some test-specific extension methods for your domain model, like this:

public static Address WithCity(this Address address, string newCity)
{
    address.City = newCity;
    return address;
}

This is same code as one of the code blocks above, only refactored to a named extension method. It simplifies use of the generic Builder, though:

var address = Build.Address().Select(a => a.WithCity("Paris")).Build();

That looks good in such a simple example, but unfortunately isn't much of an improvement when it comes to a more complex case:

var invoice =
    Build.Invoice()
        .Select(i => i
            .WithRecipient(Build.Recipient()
                .Select(r => r
                    .WithAddress(Build.Address()
                        .WithNoPostCode()
                        .Build()))
                .Build()))
        .Build();

If, at this point, you're tempted to give up on the overall strategy with a single generic Builder, you'd be excused. It will, however, turn out to be beneficial to carry on. There are more obstacles, but eventually, things will start to fall into place.

Copy and update

The above WithCity extension method mutates the input object, which can lead to surprising behaviour. While it's a common way to implement fluent interfaces in object-oriented languages, nothing prevents you from making the code saner. Instead of mutating the input object, create a new object with the single value changed:

public static Address WithCity(this Address address, string newCity)
{
    return new Address(address.Street, newCity, address.PostCode);
}

Some people will immediately be concerned about the performance implications of doing this, but you're not one of those people, are you?

Granted, there's allocation and garbage collection overhead by creating new objects like this, but I'd digress if I started to discuss this here. In most cases, the impact is insignificant.

Fluent domain model

Using extension methods enables you to use a more elegant syntax with the Select method, but there's still some maintenance overhead. If, for now, we accept such maintenance overhead, you could ask: given that we have to define and maintain all those With[...] methods, why limit them to your test code?

Would there be any harm in defining them as proper methods on your domain model?

public Address WithCity(string newCity)
{
    return new Address(this.Street, newCity, this.PostCode);
}

The above example shows the WithCity method as an instance method on the Address class. Here's the entire Address class, refactored to an immutable class:

public class Address
{
    public string Street { get; }
    public string City { get; }
    public PostCode PostCode { get; }
 
    public Address(string street, string city, PostCode postCode)
    {
        if (street == null)
            throw new ArgumentNullException(nameof(street));
        if (city == null)
            throw new ArgumentNullException(nameof(city));
        if (postCode == null)
            throw new ArgumentNullException(nameof(postCode));
 
        this.Street = street;
        this.City = city;
        this.PostCode = postCode;
    }
 
    public Address WithStreet(string newStreet)
    {
        return new Address(newStreet, this.City, this.PostCode);
    }
 
    public Address WithCity(string newCity)
    {
        return new Address(this.Street, newCity, this.PostCode);
    }
 
    public Address WithPostCode(PostCode newPostCode)
    {
        return new Address(this.Street, this.City, newPostCode);
    }
 
    public override bool Equals(object obj)
    {
        var other = obj as Address;
        if (other == null)
            return base.Equals(obj);
 
        return object.Equals(this.Street, other.Street)
            && object.Equals(this.City, other.City)
            && object.Equals(this.PostCode, other.PostCode);
    }
 
    public override int GetHashCode()
    {
        return
            this.Street.GetHashCode() ^
            this.City.GetHashCode() ^
            this.PostCode.GetHashCode();
    }
}

Technically, you could introduce instance methods like WithCity even if you kept the class itself mutable, but once you start down that path, it makes sense to make the class immutable. As Eric Evans recommends in Domain-Driven Design, modelling your domain with (immutable) Value Objects has many benefits. Such objects should also have structural equality, which is the reason that this version of Address also overrides Equals and GetHashCode.

While it looks like more work in a language like C# or Java, there are many benefits to be derived from modelling your domain with Value Objects. As an interim result, then, observe that working with unit testing (in this case a general-purpose Test Data Builder) has prompted a better design of the System Under Test.

You may still think that this seems unnecessarily verbose, and I'd agree. This is one of the many reasons I prefer languages like F# and Haskell over C# or Java. The former have such a copy and update feature built-in. Here's an F# example of updating an Address record with a specific city:

let address = { a with City = "Paris" }

This capability is built into the language. You don't have to add or maintain any code in order to be able to write code like that. Notice, even, how with is a keyword. I'm not sure about the etymology of the word with used in this context, but I find the similarity compelling.

In Haskell, it looks similar:

address = a { city = "Paris" }

In other words, domain models created from immutable Value Objects are laborious in some languages, but that only suggests a deficiency in such a language.

Default Builders as values

Now that the domain model is immutable, you can define default builders as values. Previously, to start building e.g. an Address value, you had to call the Build.Address() method. When the domain model was mutable, containing a single default value inside of a Builder would enable tests to mutate that default value. Now that domain classes are immutable, this is no longer a concern, and you can instead define test-specific default builders as values:

public static class Builder
{
    public readonly static Builder<Address> Address;
    public readonly static Builder<Invoice> Invoice;
    public readonly static Builder<InvoiceLine> InvoiceLine;
    public readonly static Builder<PostCode> PostCode;
    public readonly static Builder<PoundsShillingsPence> PoundsShillingsPence;
    public readonly static Builder<Recipient> Recipient;
 
    static Builder()
    {
        PoundsShillingsPence = new Builder<PoundsShillingsPence>(
            DomainModel.PoundsShillingsPence.Zero);
        PostCode = new Builder<PostCode>(new PostCode());
        Address =
            new Builder<Address>(new Address("""", PostCode.Build()));
        Recipient =
            new Builder<Recipient>(new Recipient("", Address.Build()));
        Invoice = new Builder<Invoice>(
            new Invoice(Recipient.Build(), new List<InvoiceLine>()));
        InvoiceLine = new Builder<InvoiceLine>(
            new InvoiceLine("", PoundsShillingsPence.Build()));
    }
 
    public static Builder<Address> WithNoPostCode(this Builder<Address> b)
    {
        return b.Select(a => a.WithPostCode(new PostCode()));
    }
}

This enables you to write expressions like this:

var address = Builder.Address.Select(a => a.WithCity("Paris")).Build();

To be clear: such a static Builder class is a Test Utility API specific to your unit tests. It would often be defined in a completely different file than the Builder<T> class, perhaps even in separate libraries.

Summary

Instead of trying to automate Test Data Builders to the letter of the original design pattern description, you can define a single, reusable, generic Builder<T> class. It enables you to achieve some of the expressivity of Test Data Builders.

If you still don't find this strategy's prospects fertile, I understand. We're not done, though. In the next article, you'll see why Select is an appropriate name for the Builder's most important method, and how it relates to good abstractions.

Next: The Builder functor.


Comments

When I found myself writing too many With() methods, I created an extension to Fody code weaving tool: Fody.With.

Basically I declare the With() methods without body implementation, and then Fody does the implementation for me. It can also convert a generic version to N overloads with an implementation per each public property.

The link about has some usage examples, that hopefully make the idea clear.

2017-08-21 12:32 UTC

C# does have Object Initializer to build "address" with specified "city", similar to F# and Haskell.

2017-08-22 12:40 UTC

Harshdeep, thank you for writing. C# object initialisers aren't the same as F# Copy and Update Record Expressions. Unless I misunderstand what you mean, when you write

var address = new Address { City = "Paris" };

address will have "Paris" as City, but all other properties, such as Street and PostCode will be null. That's not what I want. That's the problem the Test Data Builder pattern attempts to address. Test values should be populated with 'good' values, not null.

I admit that I'm not keeping up with the latest developments in C#, but if I try to use the C# object initializer syntax with an existing value, like this:

var defaultAddress =
    new Address { Street = "", PostCode = new DomainModel.PostCode(), City = "" };
var address = defaultAddress { City = "Paris" };

it doesn't compile.

I'm still on Visual Studio 2015, though, so that may be it...

2017-08-22 13:27 UTC

Aah. Now I get it. Thanks for explaining. I am from C# world and certainly not into F# yet so I missunderstood "Copy & Update Expression" with "Object Initializer".

2017-08-23 5:22 UTC

Test Data Builders in C#

Tuesday, 15 August 2017 06:20:00 UTC

A brief recap of the Test Data Builder design pattern with examples in C#.

This is the first in a series of articles about the relationship between the Test Data Builder design pattern, and the identity functor.

In 2007 Nat Pryce described the Test Data Builder design pattern. The original article is easy to read, but in case you don't want to read it, here's a quick summary, with some of Nat Pryce's examples translated to C#.

The purpose of a Test Data Builder is to make it easy to create input data (or objects) for unit tests. Imagine, for example, that for a particular test case, you need an address in Paris; no other values matter. With a Test Data Builder, you can write an expression that gives you such a value:

var address = new AddressBuilder().WithCity("Paris").Build();

The address object explicity has a City value of "Paris". Any other values are default values defined by AddressBuilder. The values are there, but when they're unimportant to a particular test case, you don't have to specify them. To paraphrase Robert C. Martin, this eliminates the irrelevant, and amplifies the essentials of the test.

Address Builder

An AddressBuilder could look like this:

public class AddressBuilder
{
    private string street;
    private string city;
    private PostCode postCode;
 
    public AddressBuilder()
    {
        this.street = "";
        this.city = "";
        this.postCode = new PostCodeBuilder().Build();
    }
 
    public AddressBuilder WithStreet(string newStreet)
    {
        this.street = newStreet;
        return this;
    }
 
    public AddressBuilder WithCity(string newCity)
    {
        this.city = newCity;
        return this;
    }
 
    public AddressBuilder WithPostCode(PostCode newPostCode)
    {
        this.postCode = newPostCode;
        return this;
    }
 
    public AddressBuilder WithNoPostcode()
    {
        this.postCode = new PostCode();
        return this;
    }
 
    public Address Build()
    {
        return new Address(this.street, this.city, this.postCode);
    }
}

The Address class is simpler than the Builder:

public class Address
{
    public string Street { getset; }
    public string City { getset; }
    public PostCode PostCode { getset; }
 
    public Address(string street, string city, PostCode postCode)
    {
        this.Street = street;
        this.City = city;
        this.PostCode = postCode;
    }
}

Clearly, this class could contain some behaviour, but in order to keep the example as simple as possible, it's only a simple Data Transfer Object.

Composition

Given that AddressBuilder is more complicated than Address itself, the benefit of the pattern may seem obscure, but one of the benefits is that Test Data Builders easily compose:

var invoice = new InvoiceBuilder()
    .WithRecipient(new RecipientBuilder()
        .WithAddress(new AddressBuilder()
            .WithNoPostcode()
            .Build())
        .Build())
    .Build();

Perhaps that looks verbose, but in general, the alternative is worse. If you didn't have a Test Utility Method, you'd have to fill in all the required data for the object:

var invoice = new Invoice(
    new Recipient("Sherlock Holmes",
        new Address("221b Baker Street",
                    "London",
                    new PostCode())),
    new List<InvoiceLine> {
        new InvoiceLine("Deerstalker Hat",
            new PoundsShillingsPence(0, 3, 10)),
        new InvoiceLine("Tweed Cape",
            new PoundsShillingsPence(0, 4, 12))});

Here, the important detail drowns in data. The post code is empty because the PostCode constructor is called without arguments. This hardly jumps out when you see it. Such code neither eliminates the irrelevant, nor amplifies the essential.

Summary

Test Data Builders are useful because they are good abstractions. They enable you to write unit tests that you can trust.

The disadvantage, as you shall see, is that in languages like C# and Java, much boilerplate code is required.

Next: Generalised Test Data Builder.


Comments

You got me to finally figure out how to post comments. :) Hope everything looks alright.

So first off, great article as always. You totally hit a subject which has been driving me nuts, personally and lately. I have been developing my first FluentAPI and have been running up against both aspects of immutability and query/command separation that you have done an excellent job of presenting here on your blog. It does seem that FluentAPI design and the builder pattern you present above deviate from these principles, so it would be great to hear a little more context and valuable insight from you on how you reconcile this. Is this perhaps a C# issue that is easily remedied in F#? Thank you in advance for any assistance and for providing such a valuable technical resource here. It's been my favorite for many years now.
2017-08-15 06:52 UTC

Mike, thank you for writing. The fluent interface that I show in this article is the most common form you see in C#. While it's not my preferred variation, I use it in this article because it's a direct translation of the style used in Nat Pryce's Java code.

Ordinarily, I prefer an immutable variant, but in C# this leads to even more boilerplate code, and I didn't want to sidetrack the topic by making this recap article more complicated than absolutely necessary.

You may be pleased to learn that future articles in this very article series will show alternatives in both C#, F#, and Haskell.

2017-08-15 07:30 UTC

Hi Mark, A while ago I made a generic builder for this exact purpose. I also made som helper extension methods, that could act as sort of an Object Mother. I quite like how it work and I have used it quite a few times. So, reading this post, I thought I'd put in a link to it, as it might be usefull to other readers.

Generic Builder with Object Mother Gist

It's all in one big gist and probably not very weel structured, but if you look at the class GenericBuilder it should be quite easily understood. The examples of extensionmethods can be seen towards the end of the file.

2017-08-15 07:42 UTC

I've used test data builders in C# just like this in the past, and couldn't decide whether I liked them or not, due to all the boilerplate.

I'm looking forward to the next few posts, thanks for doing this.

2017-08-18 11:43 UTC

Hi, Mark

In C# I starated to prefer to use a "parameterized object mother". Please take a look and tell me what out think about it: Address Object Mother Gist.

From my experience it is less and simplier code. It is also a bid easier to debug. Personally, the Object Mother is the first pattern when refactoring test data creationg and I use Fluent Test Data Builder only in more complex scenarios.

@JanD: Unfortunately, your solution would not work for immutable data structures (which I prefer).

2017-08-19 19:25 UTC

Robert, thank you for writing. I haven't seen that particular C# variation before, but it looks useful. I hope that as this article series progresses, it should become increasingly clear to the reader that the Test Data Builder pattern addresses various language deficiencies. (It has, by the way, for some time been a common criticism of design patterns in general that they are nothing but patches on language deficiencies. I don't think that I agree with that 100 percent, but I certainly understand the argument.)

Nat Pryce's original article about the Test Data Builder pattern is from 2007 with example code in Java. I don't know that much about Java, but back then, I don't think C# had optional arguments (as far as I can tell, that language feature was added in 2010). My point is that the pattern described a good way to model code given the language features that were available at the time.

As a general rule, I'm not a fan of C#'s optional argument feature (because I'm concerned what it does to forwards and backwards compatibility of my APIs), but used in the way you suggest it does look useful. Perhaps it does, indeed, address all the concerns that the Test Data Builder pattern addresses. I haven't tried it, so I can't really evaluate it (yet), but it looks like it'd be worth trying out.

My overall goal with this article series is, however, slightly different. In fact, I'm not trying to sell the Test Data Builder pattern to anyone. Rather, the point is that with better API design, and with better languages, it'd be largely redundant.

2017-08-21 06:33 UTC

Hi, Mark
Thank you for this post

I personally leverage Impromptu Interface. It could be also verbose but as you only provide meaningful data it fits to Robert C. Martin credo. And it avoids creating a lot of one-shot boilerplate code and/or noising existing classes with UT specific stuff.

2017-08-21 09:12 UTC

Romain, do you have an example of that, that you could share?

2017-08-21 09:49 UTC

Partial IAddress with City value only:

var address = new {
		    City = "Paris"
		  }.ActLike<IAddress>();
			

Partial IAddress with City value and partial IPostCode with ISO value only:

var address = new {
		    City = "Paris", 
		    PostCode = new {
				      ISO = "FR"
				   }.ActLike<IPostCode>()
		  }.ActLike<IAddress>();
			

Main drawback is verbosity but intent is pretty clear.
We could reduce nested code by splitting IAddress and IPostCode declarations but it also reduces intent: we do not care about IPostCode, we care about IAddress and IPostCode is only an implementation detail.

I heavily leverage region to cope with C# verbosity and to highlight common pattern - AAA in this case - so all this code is usually hidden in one ARRANGE region.
When I need multiple declaration I used sut (System Under Test) marker to highlight main actor.

2017-08-21 21:09 UTC

Do I understand it correctly that you'd have an interface like the following, then?

public interface IAddress
{
    string City { getset; }
}

I'm not sure that I quite follow...

2017-08-22 11:43 UTC

Mark, I tend to avoid setter in my interfaces so my domain objects usually are immutable and only expose getter.
My implementation are mainly internal which prevent them to be used directly from within UT assembly (without using InternalsVisibleTo attribute).
I have factories - which implementation are also internal - to build my objects.
I then use an IoC container to access factories and create my objects.

public interface IAddress
{
    string City { get; }
    string Street { get; }
    IPostCode PostCode { get; }
}

AddressBuilder lives in UT world so must be in another assembly to avoid noising my model.
To cope with my internal visibility constraint I have at least 2 options I can live with:

  1. Using InternalsVisibleTo attribute for my UT assembly to be able to seamlessly use my types
  2. Leveraging a test container to resolve my factory and then create my objects.
To deal with the immutable constraint I can create new ones within With methods. I can live with this too.

The main drawback remains the verbosity/burden of those methods.
Using Impromptu Interface to generate partial test data spares builder classes creation while keeping verbosity acceptable and intent clear.
Does it make sense?

2017-08-22 13:24 UTC

That helps clarify things, thank you.

I know that obviously, I could try for myself, but when you write

var address = new {
		    City = "Paris"
		  }.ActLike<IAddress>();

then what will be the value of address.PostCode?

2017-08-22 13:40 UTC
It throws an exception if accessed but live peacefully otherwise. It is why I talked about partial data.
You have to be aware of this. When your test focus on a single aspect of your class you can safely use it.
Imagine you are testing a City centric algorithm: you do not care about Street, Street number, Floor, and so on.
No need to create heavy/costly objects you can safely use a partial object which is only compliant with a part of the original interface.
The way you would have deal with if you had split IAddress interface into several parts namely IHaveACity, IHaveAStreet, ...
As it only declares what it needs to work the UT intent is pretty clear. As test builder it removes noisy stuff.
2017-08-22 14:22 UTC

Now I think I get it! Thank you for taking the time to explain.

2017-08-22 14:50 UTC

From Test Data Builders to the identity functor

Monday, 14 August 2017 11:34:00 UTC

The Test Data Builder unit testing design pattern is closely related to the identity functor.

The Test Data Builder design pattern is a valuable technique for managing data for unit testing. It enables you to express test cases in such a way that the important parts of the test case stands out in your code, while the unimportant parts disappear. It perfectly fits Robert C. Martin's definition of an abstraction:

"Abstraction is the elimination of the irrelevant and the amplification of the essential"
Not only are Test Data Builders great abstractions, but they're also eminently composable. You can use fine-grained Test Data Builders as building blocks for more complex Test Data Builders. This turns out to be more than a coincidence. In this series of articles, you'll learn how Test Data Builders are closely related to the identity functor. If you don't know what a functor is, then keep reading; you'll learn about functors as well.
  1. Test Data Builders in C#
  2. Generalised Test Data Builder
  3. The Builder functor
  4. Builder as Identity
  5. Test data without Builders
By reading these articles, you'll learn the following:
  • How to make your code easier to use in unit tests.
  • What a functor is.
  • How Test Data Builders generalise.
  • Why Test Data Builders are composable.
If you've ever struggled with defining good abstractions, learning about functors (and some related concepts) will help.

For readers wondering if this is 'yet another monad tutorial', it's not; it's a functor tutorial.

Next: Test Data Builders in C#.


F# free monad recipe

Monday, 07 August 2017 08:11:00 UTC

How to create free monads in F#.

This is not a design pattern, but it's something related. Let's call it a recipe. A design pattern should, in my opinion, be fairly language-agnostic (although hardly universally applicable). This article, on the contrary, specifically addresses a problem in F#:

How do you create a free monad in F#?

By following the present recipe.

The recipe here is a step-by-step process, but be sure to first read the sections on motivation and when to use it. A free monads isn't a goal in itself.

This article doesn't attempt to explain the details of free monads, but instead serve as a reference. For an introduction to free monads, I think my article Pure times is a good place to start. See also the Motivating examples section, below.

Motivation

A frequently asked question about F# is: what's the F# equivalent to an interface? There's no single answer to this question, because, as always, It Depends™. Why do you need an interface in the first place? What is its intended use?

Sometimes, in OOP, an interface can be used for a Strategy. This enables you to dynamically replace or select between different (sub)algorithms at run-time. If the algorithm is pure, then an idiomatic F# equivalent would be a function.

At other times, though, the person asking the question has Dependency Injection in mind. In OOP, dependencies are often modelled as interfaces with several members. Such dependencies are systematically impure, and thereby not part of functional design. If at all possible, prefer impure/pure/impure sandwiches over interactions. Sometimes, however, you'll need something that works like an interface or abstract base class. Free monads can address such situations.

In general, a free monad allows you to build a monad from any functor, but why would you want to do that? The most common reason I've encountered is exactly in order to model impure interactions in a pure manner; in other words: Dependency Injection.

Refactor interface to functor

This recipe comes in three parts:

  1. A recipe for refactoring interfaces to a functor.
  2. The core recipe for creating a monad from any functor.
  3. A recipe for adding an interpreter.
The universal recipe for creating a monad from any functor follows in a later section. In this section, you'll see how to refactor an interface to a functor.

Imagine that you have an interface that you'd like to refactor. In C# it might look like this:

public interface IFace
{
    Out1 Member1(In1 input);
    Out2 Member2(In2 input);
}

In F#, it'd look like this:

type IFace =
    abstract member Member1 : input:In1 -> Out1
    abstract member Member2 : input:In2 -> Out2

I've deliberately kept the interface vague and abstract in order to showcase the recipe instead of a particular example. For realistic examples, refer to the examples section, further down.

To refactor such an interface to a functor, do the following:

  1. Create a discriminated union. Name it after the interface name, but append the word instruction as a suffix.
  2. Make the union type generic.
  3. For each member in the interface, add a case.
    1. Name the case after the name of the member.
    2. Declare the type of data contained in the case as a pair (a two-element tuple).
    3. Declare the type of the first element in that tuple as the type of the input argument(s) to the interface member. If the member has more than one input argument, declare it as a (nested) tuple.
    4. Declare the type of the second element in the tuple as a function. The input type of that function should be the output type of the original interface member, and the output type of the function should be the generic type argument for the union type.
  4. Add a map function for the union type. I'd recommend making this function private and avoid naming it map in order to prevent naming conflicts. I usually name this function mapI, where the I stands for instruction.
  5. The map function should take a function of the type 'a -> 'b as its first (curried) argument, and a value of the union type as its second argument. It should return a value of the union type, but with the generic type argument changed from 'a to 'b.
  6. For each case in the union type, map it to a value of the same case. Copy the (non-generic) first element of the pair over without modification, but compose the function in the second element with the input function to the map function.
Following that recipe, the above interface becomes this union type:

type FaceInstruction<'a> =
| Member1 of (In1 * (Out1 -> 'a))
| Member2 of (In2 * (Out2 -> 'a))

The map function becomes:

// ('a -> 'b) -> FaceInstruction<'a> -> FaceInstruction<'b>
let private mapI f = function
    | Member1 (x, next-> Member1 (x, next >> f)
    | Member2 (x, next-> Member2 (x, next >> f)

Such a combination of union type and map function satisfies the functor laws, so that's how you refactor an interface to a functor.

Free monad recipe

Given any functor, you can create a monad. The monad will be a new type that contains the functor; you will not be turning the functor itself into a monad. (Some functors can be turned into monads themselves, but if that's the case, you don't need to create a free monad.)

The recipe for turning any functor into a monad is as follows:

  1. Create a generic discriminated union. You can name it after the underlying functor, but append a suffix such as Program. In the following, this is called the 'program' union type.
  2. Add two cases to the union: Free and Pure.
  3. The Free case should contain a single value of the contained functor, generically typed to the 'program' union type itself. This is a recursive type definition.
  4. The Pure case should contain a single value of the union's generic type.
  5. Add a bind function for the new union type. The function should take two arguments:
  6. The first argument to the bind function should be a function that takes the generic type argument as input, and returns a value of the 'program' union type as output. In the rest of this recipe, this function is called f.
  7. The second argument to the bind function should be a 'program' union type value.
  8. The return type of the bind function should be a 'program' union type value, with the same generic type as the return type of the first argument (f).
  9. Declare the bind function as recursive by adding the rec keyword.
  10. Implement the bind function by pattern-matching on the Free and Pure cases:
  11. In the Free case, pipe the contained functor value to the functor's map function, using bind f as the mapper function; then pipe the result of that to Free.
  12. In the Pure case, return f x, where x is the value contained in the Pure case.
  13. Add a computation expression builder, using bind for Bind and Pure for Return.
Continuing the above example, the 'program' union type becomes:

type FaceProgram<'a> =
| Free of FaceInstruction<FaceProgram<'a>>
| Pure of 'a

It's worth noting that the Pure case always looks like that. While it doesn't take much effort to write it, you could copy and paste it from another free monad, and no changes would be required.

According to the recipe, the bind function should be implemented like this:

// ('a -> FaceProgram<'b>) -> FaceProgram<'a> -> FaceProgram<'b>
let rec bind f = functionFree x -> x |> mapI (bind f) |> FreePure x -> f x

Apart from one small detail, the bind function always looks like that, so you can often copy and paste it from here and use it in your code, if you will. The only variation is that the underlying functor's map function isn't guaranteed to be called mapI - but if it is, you can use the above bind function as is. No modifications will be necessary.

In F#, a monad is rarely a goal in itself, but once you have a monad, you can add a computation expression builder:

type FaceBuilder () =
    member this.Bind (x, f) = bind f x
    member this.Return x = Pure x
    member this.ReturnFrom x = x
    member this.Zero () = Pure ()

While you could add more members (such as Combine, For, TryFinally, and so on), I find that usually, those four methods are all I need.

Create an instance of the builder object, and you can start writing computation expressions:

let face = FaceBuilder ()

Finally, as an optional step, if you've refactored an interface to an instruction set, you can add convenience functions that lift each instruction case to the free monad type:

  1. For each case, add a function of the same name, but camelCased instead of PascalCased.
  2. Each function should have input arguments that correspond to the first element of the case's contained tuple (i.e. the input argument for the original interface). I usually prefer the arguments in curried form, but that's not a requirement.
  3. Each function should return the corresponding instruction union case inside of the Free case. The case constructor must be invoked with the pair of data it requires. Populate the first element with values from the input arguments to the convenience function. The second element should be the Pure case constructor, passed as a function.
In the current example, that would be two functions, one for each case of FaceInstruction<'a>:

// In1 -> FaceProgram<Out1>
let member1 in1 = Free (Member1 (in1, Pure))
 
// In2 -> FaceProgram<Out2>
let member2 in2 = Free (Member2 (in2, Pure))

Such functions are conveniences that make it easier to express what the underlying functor expresses, but in the context of the free monad.

Interpreters

A free monad is a recursive type, and values are trees. The leafs are the Pure values. Often (if not always), the point of a free monad is to evaluate the tree in order to pull the leaf values out of it. In order to do that, you must add an interpreter. This is a function that recursively pattern-matches over the free monad value until it encounters a Pure case.

At least in the case where you've refactored an interface to a functor, writing an interpreter also follows a recipe. This is equivalent to writing a concrete class that implements an interface.

  1. For each case in the instruction-set functor, write an implementation function that takes the case's 'input' tuple element type as input, and returns a value of the type used in the case's second tuple element. Recall that the second element in the pair is a function; the output type of the implementation function should be the input type for that function.
  2. Add a function to implement the interpreter; I often call it interpret. Make it recursive by adding the rec keyword.
  3. Pattern-match on Pure and each case contained in Free.
  4. In the Pure case, simply return the value contained in the case.
  5. In the Free case, pattern-match the underlying pair out if each of the instruction-set functor's cases. The first element of that tuple is the 'input value'. Pipe that value to the corresponding implementation function, pipe the return value of that to the function contained in the second element of the tuple, and pipe the result of that recursively to the interpreter function.
Assume that two implementation functions imp1 and imp2 exist. According to the recipe, imp1 has the type In1 -> Out1, and imp2 has the type In2 -> Out2. Given these functions, the running example becomes:

// FaceProgram<'a> -> 'a
let rec interpret = function
    | Pure x -> x
    | Free (Member1 (x, next)) -> x |> imp1 |> next |> interpret
    | Free (Member2 (x, next)) -> x |> imp2 |> next |> interpret

The Pure case always looks like that. Each of the Free cases use a different implementation function, but apart from that, they are, as you can tell, the spitting image of each other.

Interpreters like this are often impure because the implementation functions are impure. Nothing prevents you from defining pure interpreters, although they often have limited use. They do have their place in unit testing, though.

// Out1 -> Out2 -> FaceProgram<'a> -> 'a
let rec interpretStub out1 out2 = function
    | Pure x -> x
    | Free (Member1 (_, next)) -> out1 |> next |> interpretStub out1 out2
    | Free (Member2 (_, next)) -> out2 |> next |> interpretStub out1 out2

This interpreter effectively ignores the input value contained within each Free case, and instead uses the pure values out1 and out2. This is essentially a Stub - an 'implementation' that always returns pre-defined values.

The point is that you can have more than a single interpreter, pure or impure, just like you can have more than one implementation of an interface.

When to use it

Free monads are often used instead of Dependency Injection. Note, however, that while the free monad values themselves are pure, they imply impure behaviour. In my opinion, the main benefit of pure code is that, as a code reader and maintainer, I don't have to worry about side-effects if I know that the code is pure. With a free monad, I do have to worry about side-effects, because, although the ASTs are pure, an impure interpreter will cause side-effects to happen. At least, however, the side-effects are known; they're restricted to a small subset of operations. Haskell enforces this distinction, but F# doesn't. The question, then, is how valuable you find this sort of design.

I think it still has some value, because a free monad explicitly communicates an intent of doing something impure. This intent becomes encoded in the types in your code base, there for all to see. Just as I prefer that functions return 'a option values if they may fail to produce a value, I like that I can tell from a function's return type that a delimited set of impure operations may result.

Clearly, creating free monads in F# requires some boilerplate code. I hope that this article has demonstrated that writing that boilerplate code isn't difficult - just follow the recipe. You almost don't have to think. Since a monad is a universal abstraction, once you've written the code, it's unlikely that you'll need to deal with it much in the future. After all, mathematical abstractions don't change.

Perhaps a more significant concern is how familiar free monads are to developers of a particular code base. Depending on your position, you could argue that free monads come with high cognitive overhead, or that they specifically lower the cognitive overhead.

Insights are obscure until you grasp them; after that, they become clear.

This applies to free monads as well. You have to put effort into understanding them, but once you do, you realise that they are more than a pattern. They are universal abstractions, governed by laws. Once you grok free monads, their cognitive load wane.

Consider, then, the developers who will be interacting with the free monad. If they already know free monads, or have enough of a grasp of monads that this might be their next step, then using free monads could be beneficial. On the other hand, if most developers are new to F# or functional programming, free monads should probably be avoided for the time being.

This flowchart summarises the above reflections:

Decision flowchart for whether or not to choose free monads as a design principle.

Your first consideration should be whether your context enables an impure/pure/impure sandwich. If so, there's no reason to make things more complicated than they have to be. To use Fred Brooks' terminology, this should go a long way to avoid accidental complexity.

If you can't avoid long-running, impure interactions, then consider whether purity, or strictly functional design, is important to you. F# is a multi-paradigmatic language, and it's perfectly possible to write code that's impure, yet still well-structured. You can use partial application as an idiomatic alternative to Dependency Injection.

If you prefer to keep your code functional and explicit, you may consider using free monads. In this case, I still think you should consider the maintainers of the code base in question. If everyone involved are comfortable with free monads, or willing to learn, then I believe it's a viable option. Otherwise, I'd recommend falling back to partial application, even though Dependency Injection makes everything impure.

Motivating examples

The strongest motivation, I believe, for introducing free monads into a code base is to model long-running, impure interactions in a functional style.

Like most other software design considerations, the overall purpose of application architecture is to deal with (essential) complexity. Thus, any example must be complex enough to warrant the design. There's little point in a Dependency Injection hello world example in C#. Likewise, a hello world example using free monads hardly seems justified. For that reason, examples are provided in separate articles.

A good place to start, I believe, is with the small Pure times article series. These articles show how to address a particular, authentic problem using strictly functional programming. The focus of these articles is on problem-solving, so they sometimes omit detailed explanations in order to keep the narrative moving.

If you need detailed explanations about all elements of free monads in F#, the present article series offers just that, particularly the Hello, pure command-line interaction article.

Variations

The above recipes describe the regular scenario. Variations are possible. Obviously, you can choose different naming strategies and so on, but I'm not going to cover this in greater detail.

There are, however, various degenerate cases that deserve a few words. An interaction may return no data, or take no input. In F#, you can always model the lack of data as unit (()), so it's definitely possible to define an instruction case like Foo of (unit * Out1 -> 'a), or Bar of (In2 * unit -> 'a), but since unit doesn't contain any data, you can remove it without changing the abstraction.

The Hello, pure command-line interaction article contains a single type that exemplifies both degenerate cases. It defines this instruction set:

type CommandLineInstruction<'a> =
| ReadLine of (string -> 'a)
| WriteLine of string * 'a

The ReadLine case takes no input, so instead of containing a pair of input and continuation, this case contains only the continuation function. Likewise, the WriteLine case is also degenerate, but here, there's no output. This case does contain a pair, but the second element isn't a function, but a value.

This has some superficial consequences for the implementation of functor and monad functions. For example, the mapI function becomes:

// ('a -> 'b) -> CommandLineInstruction<'a> -> CommandLineInstruction<'b>
let private mapI f = function
    | ReadLine next -> ReadLine (next >> f)
    | WriteLine (x, next) -> WriteLine (x, next |> f)

Notice that in the ReadLine case, there's no tuple on which to pattern-match. Instead, you can directly access next.

In the WriteLine case, the return value changes from function composition (next >> f) to a regular function call (next |> f, which is equivalent to f next).

The lift functions also change:

// CommandLineProgram<string>
let readLine = Free (ReadLine Pure)
 
// string -> CommandLineProgram<unit>
let writeLine s = Free (WriteLine (s, Pure ()))

Since there's no input, readLine degenerates to a value, instead of a function. On the other hand, while writeLine remains a function, you'll have to pass a value (Pure ()) as the second element of the pair, instead of the regular function (Pure).

Apart from such minor changes, the omission of unit values for input or output has little significance.

Another variation from the above recipe that you may see relates to interpreters. In the above recipe, I described how, for each instruction, you should create an implementation function. Sometimes, however, that function is only a few lines of code. When that happens, I occasionally inline the function directly in the interpreter. Once more, the CommandLineProgram API provides an example:

// CommandLineProgram<'a> -> 'a
let rec interpret = function
    | Pure x -> x
    | Free (ReadLine  next-> Console.ReadLine () |> next |> interpret
    | Free (WriteLine (s, next)) ->
        Console.WriteLine s
        next |> interpret

Here, no custom implementation functions are required, because Console.ReadLine and Console.WriteLine already exist and serve the desired purpose.

Summary

This article describes a repeatable, and automatable, process for refactoring an interface to a free monad. I've done this enough times now that I believe that this process is always possible, but I have no formal proof for this.

I also strongly suspect that the reverse process is possible. For any instruction set elevated to a free monad, I think you should be able to define an object-oriented interface. If this is true, then object-oriented interfaces and AST-based free monads are isomorphic.


Combining free monads in F#

Monday, 31 July 2017 12:30:00 UTC

An example of how to compose free monads in F#.

This article is an instalment in a series of articles about modelling long-running interactions with pure, functional code. In the previous article, you saw how to combine a pure command-line API with an HTTP-client API in Haskell. In this article, you'll see how to translate the Haskell proof of concept to F#.

HTTP API client module

You've already seen how to model command-line interactions as pure code in a previous article. You can define interactions with the online restaurant reservation HTTP API in the same way. First, define some types required for input and output to the API:

type Slot = { Date : DateTimeOffset; SeatsLeft : int }
 
type Reservation = {
    Date : DateTimeOffset
    Name : string
    Email : string
    Quantity : int }

The Slot type contains information about how many available seats are left on a particular date. The Reservation type contains the information required in order to make a reservation. It's the same Reservation F# record type you saw in a previous article, but now it's moved here.

The online restaurant reservation HTTP API may afford more functionality than you need, but there's no reason to model more instructions than required:

type ReservationsApiInstruction<'a> =
| GetSlots of (DateTimeOffset * (Slot list -> 'a))
| PostReservation of Reservation * 'a

This instruction set models two interactions. The GetSlots case models an instruction to request, from the HTTP API, the slots for a particular date. The PostReservation case models an instruction to make a POST HTTP request with a Reservation, thereby making a reservation.

While Haskell can automatically make this type a Functor, in F# you have to write the code yourself:

// ('a -> 'b) -> ReservationsApiInstruction<'a>
// -> ReservationsApiInstruction<'b>
let private mapI f = function
    | GetSlots (x, next-> GetSlots (x, next >> f)
    | PostReservation (x, next) -> PostReservation (x, next |> f)

This turns ReservationsApiInstruction<'a> into a functor, which is, however, not the ultimate goal. The final objective is to enable syntactic sugar, so that you can write pure ReservationsApiInstruction<'a> Abstract Syntax Trees (ASTs) in standard F# syntax. In order to fulfil that ambition, you need a computation expression builder, and to create one of those, you need a monad.

You can turn ReservationsApiInstruction<'a> into a monad using the free monad recipe that you've already seen. Creating a free monad, however, involves adding another type that will become both monad and functor, so I deliberately make mapI private in order to prevent confusion. This is also the reason I didn't name the function map: you'll need that name for a different type. The I in mapI stands for instruction.

The mapI function pattern-matches on the (implicit) ReservationsApiInstruction argument. In the GetSlots case, it returns a new GetSlots value, but composes the next continuation with f. In the PostReservation case, it returns a new PostReservation value, but pipes next to f. The reason for the difference is that PostReservation is degenerate: next isn't a function, but a value.

Now that ReservationsApiInstruction<'a> is a functor, you can create a free monad from it. The first step is to introduce a new type for the monad:

type ReservationsApiProgram<'a> =
| Free of ReservationsApiInstruction<ReservationsApiProgram<'a>>
| Pure of 'a

This is a recursive type that enables you to assemble ASTs that ultimately can return a value. The Pure case enables you to return a value, while the Free case lets you describe what should happen next.

Using mapI, you can make a monad out of ReservationsApiProgram<'a> by adding a bind function:

// ('a -> ReservationsApiProgram<'b>) -> ReservationsApiProgram<'a>
// -> ReservationsApiProgram<'b>
let rec bind f = function
    | Free instruction -> instruction |> mapI (bind f) |> Free
    | Pure x -> f x

If you refer back to the bind implementation for CommandLineProgram<'a>, you'll see that it's the exact same code. In Haskell, creating a free monad from a functor is automatic. In F#, it's boilerplate.

Likewise, you can make ReservationsApiProgram<'a> a functor:

// ('a -> 'b) -> ReservationsApiProgram<'a> -> ReservationsApiProgram<'b>
let map f = bind (f >> Pure)

Again, this is the same code as in the CommandLine module. You can copy and paste it. It is, however, not the same function, because the types are different.

Finally, to round off the reservations HTTP client API, you can supply functions that lift instructions to programs:

// DateTimeOffset -> ReservationsApiProgram<Slot list>
let getSlots date = Free (GetSlots (date, Pure))
 
// Reservation -> ReservationsApiProgram<unit>
let postReservation r = Free (PostReservation (r, Pure ()))

That's everything you need to create a small computation expression builder:

type ReservationsApiBuilder () =
    member this.Bind (x, f) = ReservationsApi.bind f x
    member this.Return x = Pure x
    member this.ReturnFrom x = x
    member this.Zero () = Pure ()

Create an instance of the ReservationsApiBuilder class in order to use reservationsApi computation expressions:

let reservationsApi = ReservationsApiBuilder ()

This, in total, defines a pure API for interacting with the online restaurant reservation system, including all the syntactic sugar you'll need to stay sane. As usual, some boilerplate code is required, but I'm not too worried about its maintenance overhead, as it's unlikely to change much, once you've added it. If you've followed the recipe, the API obeys the category, functor, and monad laws, so it's not something you've invented; it's an instance of a universal abstraction.

Monad stack

The addition of the above ReservationsApi module is only a step towards the overall goal, which is to write a command-line wizard you can use to make reservations against the online API. In order to do so, you must combine the two monads CommandLineProgram<'a> and ReservationsApiProgram<'a>. In Haskell, you get that combination for free via the built-in generic FreeT type, which enables you to stack monads. In F#, you have to explicitly declare the type:

type CommandLineReservationsApiT<'a> =
| Run of CommandLineProgram<ReservationsApiProgram<'a>>

This is a single-case discriminated union that stacks ReservationsApiProgram and CommandLineProgram. In this incarnation, it defines a single case called Run. The reason for this is that it enables you to follow the free monad recipe without having to do much thinking. Later, you'll see that it's possible to simplify the type.

The naming is inspired by Haskell. This type is a piece of the puzzle corresponding to Haskell's FreeT type. The T in FreeT stands for transformer, because FreeT is actually something called a monad transformer. That's not terribly important in an F# context, but that's the reason I also tagged on the T in CommandLineReservationsApiT<'a>.

FreeT is actually only a 'wrapper' around another monad. In order to extract the contained monad, you can use a function called runFreeT. That's the reason I called the F# case Run.

You can easily make your stack of monads a functor:

// ('a -> 'b) -> CommandLineProgram<ReservationsApiProgram<'a>>
// -> CommandLineProgram<ReservationsApiProgram<'b>>
let private mapStack f x = commandLine {
    let! x' = x
    return ReservationsApi.map f x' }

The mapStack function uses the commandLine computation expression to access the ReservationsApiProgram contained within the CommandLineProgram. Thanks to the let! binding, x' is a ReservationsApiProgram<'a> value. You can use ReservationsApi.map to map x' with f.

It's now trivial to make CommandLineReservationsApiT<'a> a functor as well:

// ('a -> 'b) -> CommandLineReservationsApiT<'a>
// -> CommandLineReservationsApiT<'b>
let private mapT f (Run p) = mapStack f p |> Run

The mapT function simply pattern-matches the monad stack out of the Run case, calls mapStack, and pipes the return value into another Run case.

By now, it's should be fairly clear that we're following the same recipe as before. You have a functor; make a monad out of it. First, define a type for the monad:

type CommandLineReservationsApiProgram<'a> =
| Free of CommandLineReservationsApiT<CommandLineReservationsApiProgram<'a>>
| Pure of 'a

Then add a bind function:

// ('a -> CommandLineReservationsApiProgram<'b>)
// -> CommandLineReservationsApiProgram<'a>
// -> CommandLineReservationsApiProgram<'b>
let rec bind f = function
    | Free instruction -> instruction |> mapT (bind f) |> Free
    | Pure x -> f x

This is almost the same code as the above bind function for ReservationsApi. The only difference is that the underlying map function is named mapT instead of mapI. The types involved, however, are different.

You can also add a map function:

// ('a -> 'b) -> (CommandLineReservationsApiProgram<'a>
// -> CommandLineReservationsApiProgram<'b>)
let map f = bind (f >> Pure)

This is another copy-and-paste job. Such repeatable. Wow.

When you create a monad stack, you need a way to lift values from each of the constituent monads up to the combined monad. In Haskell, this is done with the lift and liftF functions, but in F#, you must explicitly add such functions:

// CommandLineProgram<ReservationsApiProgram<'a>>
// -> CommandLineReservationsApiProgram<'a>
let private wrap x = x |> Run |> mapT Pure |> Free
// CommandLineProgram<'a> -> CommandLineReservationsApiProgram<'a>
let liftCL x = wrap <| CommandLine.map ReservationsApiProgram.Pure x
// ReservationsApiProgram<'a> -> CommandLineReservationsApiProgram<'a>
let liftRA x = wrap <| CommandLineProgram.Pure x

The private wrap function takes the underlying 'naked' monad stack (CommandLineProgram<ReservationsApiProgram<'a>>) and turns it into a CommandLineReservationsApiProgram<'a> value. It first wraps x in Run, which turns x into a CommandLineReservationsApiT<'a> value. By piping that value into mapT Pure, you get a CommandLineReservationsApiT<CommandLineReservationsApiProgram<'a>> value that you can finally pipe into Free in order to produce a CommandLineReservationsApiProgram<'a> value. Phew!

The liftCL function lifts a CommandLineProgram (CL) to CommandLineReservationsApiProgram by first using CommandLine.map to lift x to a CommandLineProgram<ReservationsApiProgram<'a>> value. It then pipes that value to wrap.

Likewise, the liftRA function lifts a ReservationsApiProgram (RA) to CommandLineReservationsApiProgram. It simply elevates x to a CommandLineProgram value by using CommandLineProgram.Pure. Subsequently, it pipes that value to wrap.

In both of these functions, I used the slightly unusual backwards pipe operator <|. The reason for that is that it emphasises the similarity between liftCL and liftRA. This is easier to see if you remove the type comments:

let liftCL x = wrap <| CommandLine.map ReservationsApiProgram.Pure x
let liftRA x = wrap <| CommandLineProgram.Pure x

This is how I normally write my F# code. I only add the type comments for the benefit of you, dear reader. Normally, when you have an IDE, you can always inspect the types using the built-in tools.

Using the backwards pipe operator makes it immediately clear that both functions depend in the wrap function. This would have been muddied by use of the normal forward pipe operator:

let liftCL x = CommandLine.map ReservationsApiProgram.Pure x |> wrap
let liftRA x = CommandLineProgram.Pure x |> wrap

The behaviour is the same, but now wrap doesn't align, making it harder to discover the kinship between the two functions. My use of the backward pipe operator is motivated by readability concerns.

Following the free monad recipe, now create a computation expression builder:

type CommandLineReservationsApiBuilder () =
    member this.Bind (x, f) = CommandLineReservationsApi.bind f x
    member this.Return x = Pure x
    member this.ReturnFrom x = x
    member this.Zero () = Pure ()

Finally, create an instance of the class:

let commandLineReservationsApi = CommandLineReservationsApiBuilder ()

Putting the commandLineReservationsApi value in a module will enable you to use it for computation expressions whenever you open that module. I normally put it in a module with the [<AutoOpen>] attribute so that it automatically becomes available as soon as I open the containing namespace.

Simplification

While there can be good reasons to introduce single-case discriminated unions in your F# code, they're isomorphic with their contained type. (This means that there's a lossless conversion between the union type and the contained type, in both directions.) Following the free monad recipe, I introduced CommandLineReservationsApiT as a discriminated union, but since it's a single-case union, you can refactor it to its contained type.

If you delete the CommandLineReservationsApiT type, you'll first have to change the definition of the program type to this:

type CommandLineReservationsApiProgram<'a> =
| Free of CommandLineProgram<ReservationsApiProgram<CommandLineReservationsApiProgram<'a>>>
| Pure of 'a

You simply replace CommandLineReservationsApiT<_> with CommandLineProgram<ReservationsApiProgram<_>>, effectively promoting the type contained in the Run case to be the container in the Free case.

Once CommandLineReservationsApiT is gone, you'll also need to delete the mapT function, and amend bind:

// ('a -> CommandLineReservationsApiProgram<'b>)
// -> CommandLineReservationsApiProgram<'a>
// -> CommandLineReservationsApiProgram<'b>
let rec bind f = function
    | Free instruction -> instruction |> mapStack (bind f) |> Free
    | Pure x -> f x

Likewise, you must also adjust the wrap function:

let private wrap x = x |> mapStack Pure |> Free

The rest of the above code stays the same.

Wizard

In Haskell, you get combinations of monads for free via the FreeT type, whereas in F#, you have to work for it. Once you have the combination in monadic form as well, you can write programs with that combination. Here's the wizard that collects your data and attempts to make a restaurant reservation on your behalf:

// CommandLineReservationsApiProgram<unit>
let tryReserve = commandLineReservationsApi {
    let! count = liftCL readQuantity
    let! date  = liftCL readDate
    let! availableSeats =
        ReservationsApi.getSlots date
        |> ReservationsApi.map (List.sumBy (fun slot -> slot.SeatsLeft))
        |> liftRA
    if availableSeats < count
    then do!
        sprintf "Only %i remaining seats." availableSeats
        |> CommandLine.writeLine
        |> liftCL
    else
        let! name  = liftCL readName
        let! email = liftCL readEmail
        do! { Date = date; Name = name; Email = email; Quantity = count }
            |> ReservationsApi.postReservation 
            |> liftRA
    }

Notice that tryReserve is a value, and not a function. It's a pure value that contains an AST - a small program that describes the impure interactions that you'd like to take place. It's defined entirely within a commandLineReservationsApi computation expression.

It starts by using the readQuantity and readDate program values you saw in the previous F# article. Both of these values are CommandLineProgram values, so you have to use liftCL to lift them to CommandLineReservationsApiProgram values - only then can you let! bind them to an int and a DateTimeOffset, respectively. This is just like the use of lift in the previous article's Haskell example.

Once the program has collected the desired date from the user, it calls ReservationsApi.getSlots and calculates the sum over all the returned SeatsLeft labels. The ReservationsApi.getSlots function returns a ReservationsApiProgram<Slot list>, the ReservationsApi.map turns it into a ReservationsApiProgram<int> value that you must liftRA in order to be able to let! bind it to an int value. Let me stress once again: the program actually doesn't do any of that; it constructs an AST with instructions to that effect.

If it turns out that there's too few seats left, the program writes that on the command line and exits. Otherwise, it continues to collect the user's name and email address. That's all the data required to create a Reservation record and pipe it to ReservationsApi.postReservation.

Interpreters

The tryReserve wizard is a pure value. It contains an AST that can be interpreted in such a way that impure operations happen. You've already seen the CommandLineProgram interpreter in a previous article, so I'm not going to repeat it here. I'll only note that I renamed it to interpretCommandLine because I want to use the name interpret for the combined interpreter.

The interpreter for ReservationsApiProgram values is similar to the CommandLineProgram interpreter:

// ReservationsApiProgram<'a> -> 'a
let rec interpretReservationsApi = function
    | ReservationsApiProgram.Pure x -> x
    | ReservationsApiProgram.Free (GetSlots (d, next)) ->
        ReservationHttpClient.getSlots d
        |> Async.RunSynchronously        
        |> next
        |> interpretReservationsApi
    | ReservationsApiProgram.Free (PostReservation (r, next)) ->
        ReservationHttpClient.postReservation r |> Async.RunSynchronously
        next |> interpretReservationsApi

The interpretReservationsApi function pattern-matches on its (implicit) ReservationsApiProgram argument, and performs the appropriate actions according to each instruction. In all Free cases, it delegates to implementations defined in a ReservationHttpClient module. The code in that module isn't shown here, but you can see it in the GitHub repository that accompanies this article.

You can combine the two 'leaf' interpreters in an interpreter of CommandLineReservationsApiProgram values:

// CommandLineReservationsApiProgram<'a> -> 'a
let rec interpret = function
    | CommandLineReservationsApiProgram.Pure x -> x
    | CommandLineReservationsApiProgram.Free p ->
        p |> interpretCommandLine |> interpretReservationsApi |> interpret

As usual, in the Pure case, it simply returns the contained value. In the Free case, p is a CommandLineProgram<ReservationsApiProgram<CommandLineReservationsApiProgram<'a>>>. Since it's a CommandLineProgram value, you can interpret it with interpretCommandLine, which returns a ReservationsApiProgram<CommandLineReservationsApiProgram<'a>>. Since that's a ReservationsApiProgram, you can pipe it to interpretReservationsApi, which then returns a CommandLineReservationsApiProgram<'a>. An interpreter exists for that type as well, namely the interpret function itself, so recursively invoke it again. In other words, interpret will keep recursing until it hits a Pure case.

Execution

Everything is now in place so that you can execute your program. This is the program's entry point:

[<EntryPoint>]
let main _ =
    interpret Wizard.tryReserve
    0 // return an integer exit code

When you run it, you'll be able to have an interaction like this:

Please enter number of diners:
4
Please enter your desired date:
2017-11-25
Please enter your name:
Mark Seemann
Please enter your email address:
mark@example.net
OK

If you want to run this code sample yourself, you're going to need an appropriate HTTP API with which you can interact. I hosted the API on my local machine, and afterwards verified that the record was, indeed, written in the reservations database.

Summary

As expected, you can combine free monads in F#, although it requires more boilerplate code than in Haskell.

Next: F# free monad recipe.


Combining free monads in Haskell

Monday, 24 July 2017 15:33:00 UTC

An example on how to compose free monads in Haskell.

In the previous article in this series on pure interactions, you saw how to write a command-line wizard in F#, using a free monad to build an Abstract Syntax Tree (AST). The example collects information about a potential restaurant reservations you'd like to make. That example, however, didn't do more than that.

For a more complete experience, you'd like your command-line interface (CLI) to not only collect data about a reservation, but actually make the reservation, using the available HTTP API. This means that you'll also need to model interaction with the HTTP API as an AST, but a different AST. Then, you'll have to figure out how to compose these two APIs into a combined API.

In order to figure out how to do this in F#, I first had to do it in Haskell. In this article, you'll see how to do it in Haskell, and in the next article, you'll see how to translate this Haskell prototype to F#. This should ensure that you get a functional F# code base as well.

Command line API

Let's make an easy start of it. In a previous article, you saw how to model command-line interactions as ASTs, complete with syntactic sugar provided by a computation expression. That took a fair amount of boilerplate code in F#, but in Haskell, it's declarative:

import Control.Monad.Trans.Free (FreeliftF)
 
data CommandLineInstruction next =
    ReadLine (String -> next)
  | WriteLine String next
  deriving (Functor)
 
type CommandLineProgram = Free CommandLineInstruction
 
readLine :: CommandLineProgram String
readLine = liftF (ReadLine id)
 
writeLine :: String -> CommandLineProgram ()
writeLine s = liftF (WriteLine s ())

This is all the code required to define your AST and make it a monad in Haskell. Contrast that with all the code you have to write in F#!

The CommandLineInstruction type defines the instruction set, and makes use of a language extension called DeriveFunctor, which enables Haskell to automatically create a Functor instance from the type.

The type alias type CommandLineProgram = Free CommandLineInstruction creates a monad from CommandLineInstruction, since Free is a Monad when the underlying type is a Functor.

The readLine value and writeLine function are conveniences that lift the instructions from CommandLineInstruction into CommandLineProgram values. These were also one-liners in F#.

HTTP client API

You can write a small wizard to collect restaurant reservation data with the CommandLineProgram API, but the new requirement is to make HTTP calls so that the CLI program actually makes the reservation against the back-end system. You could extend CommandLineProgram with more instructions, but that would be to mix concerns. It'd be more appropriate to define a new instruction set for making the required HTTP requests.

This API will send and receive more complex values than simple String values, so you can start by defining their types:

data Slot = Slot { slotDate :: ZonedTime, seatsLeft :: Int } deriving (Show)
 
data Reservation =
  Reservation { reservationDate :: ZonedTime
              , reservationName :: String
              , reservationEmail :: String
              , reservationQuantity :: Int }
              deriving (Show)

The Slot type contains information about how many available seats are left on a particular date. The Reservation type contains the information required in order to make a reservation. It's similar to the Reservation F# record type you saw in the previous article.

The online restaurant reservation HTTP API may afford more functionality than you need, but there's no reason to model more instructions than required:

data ReservationsApiInstruction next =
    GetSlots ZonedTime ([Slot-> next)
  | PostReservation Reservation next
  deriving (Functor)

This instruction set models two interactions. The GetSlots case models an instruction to request, from the HTTP API, the slots for a particular date. The PostReservation case models an instruction to make a POST HTTP request with a Reservation, thereby making a reservation.

Like the above CommandLineInstruction, this type is (automatically) a Functor, which means that we can create a Monad from it:

type ReservationsApiProgram = Free ReservationsApiInstruction

Once again, the monad is nothing but a type alias.

Finally, you're going to need the usual lifts:

getSlots :: ZonedTime -> ReservationsApiProgram [Slot]
getSlots d = liftF (GetSlots d id)
 
postReservation :: Reservation -> ReservationsApiProgram ()
postReservation r = liftF (PostReservation r ())

This is all you need to write a wizard that interleaves CommandLineProgram and ReservationsApiProgram instructions in order to create a more complex AST.

Wizard

The wizard should do the following:

  • Collect the number of diners, and the date for the reservation.
  • Query the HTTP API about availability for the requested date. If insufficient seats are available, it should exit.
  • If sufficient capacity remains, collect name and email.
  • Make the reservation against the HTTP API.
Like in the previous F# examples, you can factor some of the work that the wizard performs into helper functions. The first is one that prompts the user for a value and tries to parse it:

readParse :: Read a => String -> String -> CommandLineProgram a
readParse prompt errorMessage = do
  writeLine prompt
  l <- readLine
  case readMaybe l of
    Just dt -> return dt
    Nothing -> do
      writeLine errorMessage
      readParse prompt errorMessage

It first uses writeLine to write prompt to the command line - or rather, it creates an instruction to do so. The instruction is a pure value. No side-effects are involved until an interpreter evaluates the AST.

The next line uses readLine to read the user's input. While readLine is a CommandLineProgram String value, due to Haskell's do notation, l is a String value. You can now attempt to parse that String value with readMaybe, which returns a Maybe a value that you can handle with pattern matching. If readMaybe returns a Just value, then return the contained value; otherwise, write errorMessage and recursively call readParse again.

Like in the previous F# example, the only way to continue is to write something that readMaybe can parse. There's no other way to exit; there probably should be an option to quit, but it's not important for this demo purpose.

You may also have noticed that, contrary to the previous F# example, I here succumbed to the temptation to break the rule of three. It's easier to define a reusable function in Haskell, because you can leave it generic, with the proviso that the generic value must be an instance of the Read typeclass.

The readParse function returns a CommandLineProgram a value. It doesn't combine CommandLineProgram with ReservationsApiProgram. That's going to happen in another function, but before we look at that, you're also going to need another little helper:

readAnything :: String -> CommandLineProgram String
readAnything prompt = do
  writeLine prompt
  readLine

The readAnything function simply writes a prompt, reads the user's input, and unconditionally returns it. You could also have written it as a one-liner like readAnything prompt = writeLine prompt >> readLine, but I find the above code more readable, even though it's slightly more verbose.

That's all you need to write the wizard:

tryReserve :: FreeT ReservationsApiProgram CommandLineProgram ()
tryReserve = do
  q <- lift $ readParse "Please enter number of diners:" "Not an Integer."
  d <- lift $ readParse "Please enter your desired date:" "Not a date."
  availableSeats <- liftF $ (sum . fmap seatsLeft) <$> getSlots d
  if availableSeats < q
    then lift $ writeLine $ "Only " ++ show availableSeats ++ " remaining seats."
    else do
      n <- lift $ readAnything "Please enter your name:"
      e <- lift $ readAnything "Please enter your email address:"
      liftF $ postReservation Reservation
        { reservationDate = d
        , reservationName = n
        , reservationEmail = e
        , reservationQuantity = q }

The tryReserve program first prompt the user for a number of diners and a date. Once it has the date d, it calls getSlots and calculates the sum of the remaining seats. availableSeats is an Int value like q, so you can compare those two values with each other. If the number of available seats is less than the desired quantity, the program writes that and exits.

This interaction demonstrates how to interleave CommandLineProgram and ReservationsApiProgram instructions. It would be a bad user experience if the program would ask the user to input all information, and only then discover that there's insufficient capacity.

If, on the other hand, there's enough remaining capacity, the program continues collecting information from the user, by prompting for the user's name and email address. Once all data is collected, it creates a new Reservation value and invokes postReservation.

Consider the type of tryReserve. It's a combination of CommandLineProgram and ReservationsApiProgram, contained within a type called FreeT. This type is also a Monad, which is the reason the do notation still works. This also begins to explain the various lift and liftF calls sprinkled over the code.

Whenever you use a <- arrow to 'pull the value out of the monad' within a do block, the right-hand side of the arrow must have the same type as the return type of the overall function (or value). In this case, the return type is FreeT ReservationsApiProgram CommandLineProgram (), whereas readParse returns a CommandLineProgram a value. As an example, lift turns CommandLineProgram Int into FreeT ReservationsApiProgram CommandLineProgram Int.

The way the type of tryReserve is declared, when you have a CommandLineProgram a value, you use lift, but when you have a ReservationsApiProgram a, you use liftF. This depends on the order of the monads contained within FreeT. If you swap CommandLineProgram and ReservationsApiProgram, you'll also need to use lift instead of liftF, and vice versa.

Interpreters

tryReserve is a pure value. It's an Abstract Syntax Tree that combines two separate instruction sets to describe a complex interaction between user, command line, and an HTTP client. The program doesn't do anything until interpreted.

You can write an impure interpreter for each of the APIs, and a third one that uses the other two to interpret tryReserve.

Interpreting CommandLineProgram values is similar to the previous F# example:

interpretCommandLine :: CommandLineProgram a -> IO a
interpretCommandLine program =
  case runFree program of
    Pure r -> return r
    Free (ReadLine next) -> do
      line <- getLine
      interpretCommandLine $ next line
    Free (WriteLine line next) -> do
      putStrLn line
      interpretCommandLine next

This interpreter is a recursive function that pattern-matches all the cases in any CommandLineProgram a. When it encounters a Pure case, it simply returns the contained value.

When it encounters a ReadLine value, it calls getLine, which returns an IO String value read from the command line, but thanks to the do block, line is a String value. The interpreter then calls next with line, and passes the return value of that recursively to itself.

A similar treatment is given to the WriteLine case. putStrLn line writes line to the command line, where after next is used as an input argument to interpretCommandLine.

Thanks to Haskell's type system, you can easily tell that interpretCommandLine is impure, because for every CommandLineProgram a it returns IO a. That was the intent all along.

Likewise, you can write an interpreter for ReservationsApiProgram values:

interpretReservationsApi :: ReservationsApiProgram a -> IO a
interpretReservationsApi program =
  case runFree program of
    Pure x -> return x
    Free (GetSlots zt next) -> do
      slots <- HttpClient.getSlots zt
      interpretReservationsApi $ next slots
    Free (PostReservation r next) -> do
      HttpClient.postReservation r
      interpretReservationsApi next

The structure of interpretReservationsApi is similar to interpretCommandLine. It delegates its implementation to an HttpClient module that contains the impure interactions with the HTTP API. This module isn't shown in this article, but you can see it in the GitHub repository that accompanies this article.

From these two interpreters, you can create a combined interpreter:

interpret :: FreeT ReservationsApiProgram CommandLineProgram a -> IO a
interpret program = do
  r <- interpretCommandLine $ runFreeT program
  case r of
    Pure x -> return x
    Free p -> do
      y <- interpretReservationsApi p
      interpret y

This function has the required type: it evaluates any FreeT ReservationsApiProgram CommandLineProgram a and returns an IO a. runFreeT returns the CommandLineProgram part of the combined program. Passing this value to interpretCommandLine, you get the underlying type - the a in CommandLineProgram a, if you will. In this case, however, the a is quite a complex type that I'm not going to write out here. Suffice it to say that, at the container level, it's a FreeF value, which can be either a Pure or a Free case that you can use for pattern matching.

In the Pure case, you're done, so you can simply return the underlying value.

In the Free case, the p contained inside is a ReservationsApiProgram value, which you can interpret with interpretReservationsApi. That returns an IO a value, and due to the do block, y is the a. In this case, however, a is FreeT ReservationsApiProgram CommandLineProgram a, but that means that the function can now recursively call itself with y in order to interpret the next instruction.

Execution

Armed with both an AST and an interpreter, executing the program is trivial:

main :: IO ()
main = interpret tryReserve

When you run the program, you could produce an interaction like this:

Please enter number of diners:
4
Please enter your desired date:
2017-11-25 18-30-00Z
Not a date.
Please enter your desired date:
2017-11-25 18:30:00Z
Please enter your name:
Mark Seemann
Please enter your email address:
mark@example.org
Status {statusCode = 200, statusMessage = "OK"}

You'll notice that I initially made a mistake on the date format, which caused readParse to prompt me again.

If you want to run this code sample yourself, you're going to need an appropriate HTTP API with which you can interact. I hosted the API on my local machine, and afterwards verified that the record was, indeed, written in the reservations database.

Summary

This proof of concept proves that it's possible to combine separate free monads. Now that we know that it works, and the overall outline of it, it should be possible to translate this to F#. You should, however, expect more boilerplate code.

Next: Combining free monads in F#.


Comments

Here's an additional simplification. Rather than writing FreeT ReservationsApiProgram CommandLineProgram which requires you to lift, you can instead form the sum (coproduct) of both functors:

import Data.Functor.Sum

type Program = Free (Sum CommandLineInstruction ReservationsApiInstruction)

liftCommandLine :: CommandLineInstruction a -> Program a
liftCommandLine = liftF . InL

liftReservation :: ReservationsApiInstruction a -> Program a
liftReservation = liftF . InR

Now you can lift the helpers directly to Program, like so:

readLine :: Program String
readLine = liftCommandLine (ReadLine id)
 
writeLine :: String -> Program ()
writeLine s = liftCommandLine (WriteLine s ())

getSlots :: ZonedTime -> Program [Slot]
getSlots d = liftReservation (GetSlots d id)
 
postReservation :: Reservation -> Program ()
postReservation r = liftReservation (PostReservation r ())

Then (after you change the types of the read* helpers), you can drop all lifts from tryReserve:

tryReserve :: Program ()
tryReserve = do
  q <- readParse "Please enter number of diners:" "Not an Integer."
  d <- readParse "Please enter your desired date:" "Not a date."
  availableSeats <- (sum . fmap seatsLeft) <$> getSlots d
  if availableSeats < q
    then writeLine $ "Only " ++ show availableSeats ++ " remaining seats."
    else do
      n <- readAnything "Please enter your name:"
      e <- readAnything "Please enter your email address:"
      postReservation Reservation
        { reservationDate = d
        , reservationName = n
        , reservationEmail = e
        , reservationQuantity = q }

And finally your interpreter needs to dispatch over InL/InR (this is using functions from Control.Monad.Free, you can actually drop the Trans import at this point):

interpretCommandLine :: CommandLineInstruction (IO a) -> IO a
interpretCommandLine (ReadLine next) = getLine >>= next
interpretCommandLine (WriteLine line next) = putStrLn line >> next

interpretReservationsApi :: ReservationsApiInstruction (IO a) -> IO a
interpretReservationsApi (GetSlots zt next) = HttpClient.getSlots zt >>= next
interpretReservationsApi (PostReservation r next) = HttpClient.postReservation r >> next

interpret :: Program a -> IO a
interpret program =
  iterM go program
  where
    go (InL cmd) = interpretCommandLine cmd
    go (InR res) = interpretReservationsApi res

I find this to be quite clean!

2017-07-27 3:58 UTC

George, thank you for writing. That alternative does, indeed, look simpler and cleaner than mine. Thank you for sharing.

FWIW, one reason I write articles on this blog is to learn and become better. I publish what I know and have learned so far, and sometimes, people tell me that there's a better way. That's great, because it makes me a better programmer, and hopefully, it may make other readers better as well.

In case you'll be puzzling over my next blog post, however, I'm going to share a little secret (which is not a secret if you look at the blog's commit history): I wrote this article series more than a month ago, which means that all the remaining articles are already written. While I agree that using the sum of functors instead of FreeT simplifies the Haskell code, I don't think it makes that much of a difference when translating to F#. I may be wrong, but I haven't tried yet. My point, though, is that the next article in the series is going to ignore this better alternative, because, when it was written, I didn't know about it. I invite any interested reader to post, as a comment to that future article, their better alternatives :)

2017-07-27 7:31 UTC

Hi Mark,

I think you'll enjoy Data Types a la Carte. It's the definitive introduction to the style that George Pollard demonstrates above. Swierstra covers how to build datatypes with initial algebras over coproducts, compose them abstracting over the concrete functor, and tear them down generically. It's well written, too 😉

Benjamin

2017-07-23 28:40 UTC

A pure command-line wizard

Monday, 17 July 2017 12:04:00 UTC

An example of a small Abstract Syntax Tree written with F# syntactic sugar.

In the previous article, you got an introduction to a functional command-line API in F#. The example in that article, however, was too simple to highlight its composability. In this article, you'll see a fuller example.

Command-line wizard for on-line restaurant reservations

In previous articles, you can see variations on an HTTP-based back-end for an on-line restaurant reservation system. In this article, on the other hand, you're going to see a first attempt at a command-line client for the API.

Normally, an on-line restaurant reservation system would have GUIs hosted in web pages or mobile apps, but with an open HTTP API, a self-respecting geek would prefer a command-line interface (CLI)... right?!

Please enter number of diners:
four
Not an integer.
Please enter number of diners:
4
Please enter your desired date:
My next birthday
Not a date.
Please enter your desired date:
2017-11-25
Please enter your name:
Mark Seemann
Please enter your email address:
mark@example.com
{Date = 25.11.2017 00:00:00 +01:00;
 Name = "Mark Seemann";
 Email = "mark@example.com";
 Quantity = 4;}

In this incarnation, the CLI only collects information in order to dump a rendition of an F# record on the command-line. In a future article, you'll see how to combine this with an HTTP client in order to make a reservation with the back-end system.

Notice that the CLI is a wizard. It leads you through a series of questions. You have to give an appropriate answer to each question before you can move on to the next question. For instance, you must type an integer for the number of guests; if you don't, the wizard will repeatedly ask you for an integer until you do.

You can develop such an interface with the commandLine computation expression from the previous article.

Reading quantities

There are four steps in the wizard. The first is to read the desired quantity from the command line:

// CommandLineProgram<int>
let rec readQuantity = commandLine {
    do! CommandLine.writeLine "Please enter number of diners:"
    let! l = CommandLine.readLine
    match Int32.TryParse l with
    | true, dinerCount -> return dinerCount
    | _ ->
        do! CommandLine.writeLine "Not an integer."
        return! readQuantity }

This small piece of interaction is defined entirely within a commandLine expression. This enables you to use do! expressions and let! bindings to compose smaller CommandLineProgram values, such as CommandLine.writeLine and CommandLine.readLine (both shown in the previous article).

After prompting the user to enter a number, the program reads the user's input from the command line. While CommandLine.readLine is a CommandLineProgram<string> value, the let! binding turns l into a string value. If you can parse l as an integer, you return the integer; otherwise, you recursively return readQuantity.

The readQuantity program will continue to prompt the user for an integer. It gives you no option to cancel the wizard. This is a deliberate simplification I did in order to keep the example as simple as possible, but a real program should offer an option to abort the wizard.

The function returns a CommandLineProgram<int> value. This is a pure value containing an Abstract Syntax Tree (AST) that describes the interactions to perform. It doesn't do anything until interpreted. Contrary to designing with Dependency Injection and interfaces, however, you can immediately tell, from the type, that explicitly delimited impure interactions may take place within that part of your code base.

Reading dates

When you've entered a proper number of diners, you proceed to enter a date. The program for that looks similar to readQuantity:

// CommandLineProgram<DateTimeOffset>
let rec readDate = commandLine {
    do! CommandLine.writeLine "Please enter your desired date:"
    let! l = CommandLine.readLine
    match DateTimeOffset.TryParse l with
    | true, dt -> return dt
    | _ ->
        do! CommandLine.writeLine "Not a date."
        return! readDate }

The readDate value is so similar to readQuantity that you might be tempted to refactor both into a single, reusable function. In this case, however, I chose to stick to the rule of three.

Reading strings

Reading the customer's name and email address from the command line is easy, as no parsing is required:

// CommandLineProgram<string>
let readName = commandLine {
    do! CommandLine.writeLine "Please enter your name:"
    return! CommandLine.readLine }
 
// CommandLineProgram<string>
let readEmail = commandLine {
    do! CommandLine.writeLine "Please enter your email address:"
    return! CommandLine.readLine }

Both of these values unconditionally accept whatever you write when prompted. From a security standpoint, all input is evil, so in a production code base, you should still perform some validation. This, on the other hand, is demo code, so with that caveat, it accepts all strings you might type.

These values are similar to each other, but once again I invoke the rule of three and keep them as separate values.

Composing the wizard

Together with the general-purpose command line API, the above values are all you need to compose the wizard. In this incarnation, the wizard should collect the information you type, and create a single record with those values. This is the type of record it must create:

type Reservation = {
    Date : DateTimeOffset
    Name : string
    Email : string
    Quantity : int }

You can easily compose the wizard like this:

// CommandLineProgram<Reservation>
let readReservationRequest = commandLine {
    let! count = readQuantity
    let! date  = readDate
    let! name  = readName
    let! email = readEmail
    return { Date = date; Name = name; Email = email; Quantity = count } }

There's really nothing to it. As all the previous code examples in this article, you compose the readReservationRequest value entirely inside a commandLine expression. You use let! bindings to collect the four data elements you need, and once you have all four, you can return a Reservation value.

Running the program

You may have noticed that no code so far shown define functions; they are all values. They are small program fragments, expressed as ASTs, composed into slightly larger programs that are still ASTs. So far, all the code is pure.

In order to run the program, you need an interpreter. You can reuse the interpreter from the previous article when composing your main function:

[<EntryPoint>]
let main _ =
    Wizard.readReservationRequest
    |> CommandLine.bind (CommandLine.writeLine << (sprintf "%A"))
    |> interpret
    0 // return an integer exit code

Notice that most of the behaviour is defined by the above Wizard.readReservationRequest value. That program, however, returns a Reservation value that you should also print to the command line, using the CommandLine module. You can achieve that behaviour by composing Wizard.readReservationRequest with CommandLine.writeLine using CommandLine.bind. Another way to write the same composition would be by using a commandLine computation expression, but in this case, I find the small pipeline of functions easier to read.

When you bind two CommandLineProgram values to each other, the result is a third CommandLineProgram. You can pipe that to interpret in order to run the program. The result is an interaction like the one shown in the beginning of this article.

Summary

In this article, you've seen how you can create larger ASTs from smaller ASTs, using the syntactic sugar that F# computation expressions afford. The point, so far, is that you can make side-effects and non-deterministic behaviour explicit, while retaining the 'normal' F# development experience.

In Haskell, impure code can execute within an IO context, but inside IO, any sort of side-effect or non-deterministic behaviour could take place. For that reason, even in Haskell, it often makes sense to define an explicitly delimited set of impure operations. In the previous article, you can see a small Haskell code snippet that defines a command-line instruction AST type using Free. When you, as a code reader, encounter a value of the type CommandLineProgram String, you know more about the potential impurities than if you encounter a value of the type IO String. The same argument applies, with qualifications, in F#.

When you encounter a value of the type CommandLineProgram<Reservation>, you know what sort of impurities to expect: the program will only write to the command line, or read from the command line. What if, however, you'd like to combine those particular interactions with other types of interactions?

Read on.

Next: Combining free monads in Haskell.


Hello, pure command-line interaction

Tuesday, 11 July 2017 12:48:00 UTC

A gentle introduction to modelling impure interactions with pure code.

Dependency Injection is a well-described concept in object-oriented programming, but as I've explained earlier, its not functional, because it makes everything impure. In general, you should reject the notion of dependencies by instead designing your application on the concept of an impure/pure/impure sandwich. This is possible more often than you'd think, but there's still a large group of applications where this will not work. If your application needs to interact with the impure world for an extended time, you need a way to model such interactions in a pure way.

This article introduces a way to do that.

Command line API

Imagine that you have to write a command-line program that can ask a series of questions and print appropriate responses. In the general case, this is a (potentially) long-running series of interactions between the user and the program. To keep it simple, though, in this article we'll start by looking at a degenerate example:

Please enter your name.
Mark
Hello, Mark!

The program is simply going to request that you enter your name. Once you've done that, it prints the greeting. In object-oriented programming, using Dependency Injection, you might introduce an interface. Keeping it simple, you can restrict such an interface to two methods:

public interface ICommandLine
{
    string ReadLine();
    void WriteLine(string text);
}

Please note that this is clearly a toy example. In later articles, you'll see how to expand the example to cover some more complex interactions, but you could also read a more realistic example already. Initially, the example is degenerate, because there's only a single interaction. In this case, an impure/pure/impure sandwich is still possible, but such a design wouldn't scale to more complex interactions.

The problem with defining and injecting an interface is that it isn't functional. What's the functional equivalent, then?

Instruction set

Instead of defining an interface, you can define a discriminated union that describes a limited instruction set for command-line interactions:

type CommandLineInstruction<'a> =
| ReadLine of (string -> 'a)
| WriteLine of string * 'a

You may notice that it looks a bit like the above C# interface. Instead of defining two methods, it defines two cases, but the names are similar.

The ReadLine case is an instruction that an interpreter can evaluate. The data contained in the case is a continuation function. After evaluating the instruction, an interpreter must invoke this function with a string. It's up to the interpreter to figure out which string to use, but it could, for example, come from reading an input string from the command line. The continuation function is the next step in whatever program you're writing.

The WriteLine case is another instruction for interpreters. The data contained in this case is a tuple. The first element of the tuple is input for the interpreter, which can choose to e.g. print the value on the command line, or ignore it, and so on. The second element of the tuple is a value used to continue whatever program this case is a part of.

This enables you to write a small, specialised Abstract Syntax Tree (AST), but there's currently no way to return from it. One way to do that is to add a third 'stop' case. If you're interested in that option, Scott Wlaschin covers this as one iteration in his excellent explanation of the AST design.

Instead of adding a third 'stop' case to CommandLineInstruction<'a>, another option is to add a new wrapper type around it:

type CommandLineProgram<'a> =
| Free of CommandLineInstruction<CommandLineProgram<'a>>
| Pure of 'a

The Free case contains a CommandLineInstruction that always continues to a new CommandLineProgram value. The only way you can escape the AST is via the Pure case, which simply contains the 'return' value.

Abstract Syntax Trees

With these two types you can write specialised programs that contain instructions for an interpreter. Notice that the types are pure by intent, although in F# we can't really tell. You can, however, repeat this exercise in Haskell, where the instruction set looks like this:

data CommandLineInstruction next =
    ReadLine (String -> next)
  | WriteLine String next
  deriving (Functor)
 
type CommandLineProgram = Free CommandLineInstruction

Both of these types are pure, because IO is nowhere in sight. In Haskell, functions are pure by default. This also applies to the String -> next function contained in the ReadLine case.

Back in F# land, you can write an AST that implements the command-line interaction shown in the beginning of the article:

// CommandLineProgram<unit>
let program =
    Free (WriteLine (
            "Please enter your name.",
            Free (ReadLine (
                    fun s -> Free (WriteLine (
                                    sprintf "Hello, %s!" s,
                                    Pure ()))))))

This AST defines a little program. The first step is a WriteLine instruction with the input value "Please enter your name.". The WriteLine case constructor takes a tuple as input argument. The first tuple element is that prompt, and the second element is the continuation, which has to be a new CommandLineInstruction<CommandLineProgram<'a>> value.

In this example, the continuation value is a ReadLine case, which takes a continuation function as input. This function should return a new program value, which it does by returning a WriteLine.

This second WriteLine value creates a string from the outer value s. The second tuple element for the WriteLine case must, again, be a new program value, but now the program is done, so you can use the 'stop' value Pure ().

You probably think that I should quit the mushrooms. No one in their right mind will want to write code like this. Neither would I. Fortunately, you can make the coding experience much better, but you'll see how to do that later.

Interpretation

The above program value is a small CommandLineProgram<unit>. It's a pure value. In itself, it doesn't do anything.

Clearly, we'd like it to do something. In order to make that happen, you can write an interpreter:

// CommandLineProgram<'a> -> 'a
let rec interpret = function
    | Pure x -> x
    | Free (ReadLine  next-> Console.ReadLine () |> next |> interpret
    | Free (WriteLine (s, next)) ->
        Console.WriteLine s
        next |> interpret

This interpreter is a recursive function that pattern-matches all the cases in any CommandLineProgram<'a>. When it encounters a Pure case, it simply returns the contained value.

When it encounters a ReadLine value, it calls Console.ReadLine (), which returns a string value read from the command line. It then pipes that input value to its next continuation function, which produces a new CommandLineInstruction<CommandLineProgram<'a>> value. Finally, it pipes that continuation value recursively to itself.

A similar treatment is given to the WriteLine case. Console.WriteLine s writes s to the command line, where after next is recursively piped to interpret.

When you run interpret program, you get an interaction like this:

Please enter your name.
ploeh
Hello, ploeh!

The program is pure; the interpret function is impure.

Syntactic sugar

Clearly, you don't want to write programs as ASTs like the above. Fortunately, you don't have to. You can add syntactic sugar in the form of computation expressions. The way to do that is to turn your AST types into a monad. In Haskell, you'd already be done, because Free is a monad. In F#, some code is required.

Source functor

The first step is to define a map function for the underlying instruction set union type. Conceptually, when you can define a map function for a type, you've created a functor (if it obeys the functor laws, that is). Functors are common, so it often pays off being aware of them.

// ('a -> 'b) -> CommandLineInstruction<'a> -> CommandLineInstruction<'b>
let private mapI f = function
    | ReadLine next -> ReadLine (next >> f)
    | WriteLine (x, next) -> WriteLine (x, next |> f)

The mapI function takes a CommandLineInstruction<'a> value and maps it to a new value by mapping the 'underlying value'. I decided to make the function private because later, I'm also going to define a map function for CommandLineProgram<'a>, and I don't want to confuse users of the API with two different map functions. This is also the reason that the name of the function isn't simply map, but rather mapI, where the I stands for instruction.

mapI pattern-matches on the (implicit) input argument. If it's a ReadLine case, it returns a new ReadLine value, but it uses the mapper function f to translate the next function. Recall that next is a function of the type string -> 'a. When you compose it with f (which is a function of the type 'a -> 'b), you get (string -> 'a) >> ('a -> 'b), or string -> 'b. You've transformed the 'a to a 'b for the ReadLine case. If you can do the same for the WriteLine case, you'll have a functor.

Fortunately, the WriteLine case is similar, although a small tweak is required. This case contains a tuple of data. The first element (x) isn't a generic type (it's a string), so there's nothing to map. You can use it as-is in the new WriteLine value that you'll return. The WriteLine case is degenerate because next isn't a function, but rather a value. It has a type of 'a, and f is a function of the type 'a -> 'b, so piping next to f returns a 'b.

That's it: now you have a functor.

(In order to keep the category theorists happy, I should point out that such functors are really a sub-type of functors called endo-functors. Additionally, functors must obey some simple and intuitive laws in order to be functors, but that's all I'll say about that here.)

Free monad

There's a reason I spend so much time talking about functors. The goal is syntactic sugar. You can get that with computation expressions. In order to create a computation expression builder, you need a monad.

You need a recipe for creating a monad. Fortunately, there's a type of monad called a free monad. It has the virtue that it enables you to create a monad from any functor.

Just what you need.

In Haskell, this happens automatically when you declare type CommandLineProgram = Free CommandLineInstruction. Thanks to Haskell's type system, Free is automatically a Monad when the underlying type is a Functor. In F#, you have to work for your monads, but the fact that Haskell can automate this means that there's a recipe that you can follow.

Earlier in this article, I mentioned in passing that there are alternative ways in which you can define a 'stop' case for your instruction set. The reason I chose to separate the API into two types (an 'instruction set', and a 'program') is that the instruction set is the underlying functor. The 'program' is part of the free monad. The other part is a bind function (that obeys the monad laws).

// ('a -> CommandLineProgram<'b>) -> CommandLineProgram<'a>
// -> CommandLineProgram<'b>
let rec bind f = function
    | Free instruction -> instruction |> mapI (bind f) |> Free
    | Pure x -> f x

This recursive function pattern-matches on the (implicit) CommandLineProgram<'a> argument. In the Pure case, the 'return' value x has the type 'a, which fits as input for the f function. The result is a value of type CommandLineProgram<'b>.

In the Free case, the instruction is a functor with the map function mapI. The first argument to the mapI function must be a function with the type 'a -> 'b. How can you compose such a function?

If you partially apply the recursive bind function with f (that is: bind f), you get a function of the type CommandLineProgram<'a> -> CommandLineProgram<'b>. This fits with mapI, because instruction has the type CommandLineInstruction<CommandLineProgram<'a>> (refer back to the definition of the Free case if need to convince yourself of that). The result of calling mapI with instruction is a CommandLineInstruction<CommandLineProgram<'b>> value. In order to turn it into a CommandLineProgram<'b> value, you wrap it in a new Free case.

Although this required a bit of explanation, defining a bind function for a free monad is a repeatable process. After all, in Haskell it's automated. In F#, you have to explicitly write the code, but it follows a recipe. Once you get the hang of it, there's not much to it.

Functor

You'll occasionally need to explicitly use the bind function, but often it'll 'disappear' into a computation expression. There are other building blocks to an API than a bind function, though. You may, for example, need a map function:

// ('a -> 'b) -> CommandLineProgram<'a> -> CommandLineProgram<'b>
let map f = bind (f >> Pure)

This makes CommandLineProgram<'a> a functor as well. This is the reason I made mapI private, because mapI makes the instruction set a functor, but the API is expressed in terms of AST programs, and it should be consistent. Within the same module, map should work on the same data type as bind.

Notice that map can be defined as a composition of bind and Pure. This is part of the recipe. For a free monad, the map function always looks like that. The f function is a function with the type 'a -> 'b, and Pure is a case constructor with the type 'b -> CommandLineProgram<'b>. Notice that I've used 'b for the generic type argument instead of the usual 'a. Hopefully, this makes it clear that when you compose these two functions together (f >> Pure), you get a function of the type ('a -> 'b) >> ('b -> CommandLineProgram<'b>), or 'a -> CommandLineProgram<'b>. That's just the type of function needed for the bind function, so the whole composition turns out to type-check and work as intended.

API

In order to work with an API, you need the ability to create values of the API's type(s). In this case, you must be able to create CommandLineProgram<'a> values. While you can create them explicitly using the ReadLine, WriteLine, Free, and Pure case constructors, it'll be more convenient if you have some predefined functions and values for that.

// CommandLineProgram<string>
let readLine = Free (ReadLine Pure)
 
// string -> CommandLineProgram<unit>
let writeLine s = Free (WriteLine (s, Pure ()))

In the ReadLine case, there's no input to the instruction, so you can define readLine as a predefined CommandLineProgram<string> value.

The WriteLine case, on the other hand, takes as an input argument a string to write, so you can define writeLine as a function that returns a CommandLineProgram<unit> value.

Computation expression

The addition of map and supporting API is, to be honest, a bit of digression. You're going to use these functions later, but they aren't required in order to create a computation expression builder. All you need is a bind function and a way to lift a raw value into the monad. All of these are in place, so the builder is a matter of delegation:

type CommandLineBuilder () =
    member this.Bind (x, f) = CommandLine.bind f x
    member this.Return x = Pure x
    member this.ReturnFrom x = x
    member this.Zero () = Pure ()

This is a fairly minimal builder, but in my experience, most of times, this is all you need.

Create an instance of the CommandLineBuilder class, and you can write computation expressions:

let commandLine = CommandLineBuilder ()

I usually put such an object in a module with an [<AutoOpen>] attribute, so that it's available as a global object.

Producing ASTs with pretty code

Using the commandLine computation expression is like using the built-in async or seq expressions. You can use it to rewrite the above AST as readable code:

// CommandLineProgram<unit>
let program =
    commandLine {
        do!  CommandLine.writeLine "Please enter your name."
        let! name = CommandLine.readLine
        do!  sprintf "Hello, %s!" name |> CommandLine.writeLine }

This produces the same AST as before, but with much more readable syntax. The AST is the same, and you can use the above interpret function to run it. The interaction is the same as before:

Please enter your name.
Free
Hello, Free!

This is, obviously, a toy example, but in coming articles, you'll see how to gradually enhance the code to perform some more complex interactions.

Summary

Functional programming emphasises pure functions, and a separation of pure and impure code. The simplest way to achieve such a separation is to design your code as an impure/pure/impure sandwich, but sometimes this isn't possible. When it's not possible, an alternative is to define an instruction set for an AST, and turn it into a free monad in order to enable enough syntactic sugar to keep the code readable.

While this may seem complicated, it has the benefit of making impurities explicit in the code. Whenever you see a CommandLineProgram value, you know that, at run-time, something impure is likely to happen. It's not uncontrolled impurity, though. Inside a CommandLineProgram, only reading from, and writing to, the command line will happen. It's not going to generate random values, change global variables, send an email, or any other unpredictable operation - that is, unless the interpreter does that...

Next: A pure command-line wizard.


Pure interactions

Monday, 10 July 2017 14:29:00 UTC

Long-running, non-deterministic interactions can be modelled in a pure, functional way.

In a previous article, you can read why Dependency Injection and (strict) functional programming are mutually exclusive. Dependency Injection makes everything impure, and if nothing is pure, then it's hardly functional. In Dependency rejection, you can see how you can often separate impure and pure code into an impure/pure/impure sandwich.

Micro-operation-based architectures

The impure/pure/impure sandwich architecture works well in scenarios with limited interaction. Some data arrives at the boundary of the system, the system responds, and that's it. That, however, describes a significant fraction of all software running in the world today.

Any HTTP-based application (web site, REST API, most SOAP services) fits the description: an HTTP request arrives, and the server responds with an HTTP response. In a well-designed and well-running system, you should return the response within seconds, if not faster. Everything the software needs in order to run to completion is either part of the request, or part of the application state. You may need to query a database to gather more data based on the incoming request, but you can still gather most data from impure sources, pass it all to your pure core implementation, get the pure values back and return the response.

Likewise, asynchronous message-based systems, such as pub/sub, Pipes and Filters, Actor-based systems, 'SOA done right', CQRS/Event Sourcing, and so on, are based on short-lived, stateless interactions. Similar to HTTP-based applications, there's often (persisted) application state, but once a message arrives at a message handler, the software should process it as quickly as possible. Again, it can read extra (impure) data from a database, pass everything to a pure function, and finally do something impure with the return value.

Common for all such systems is that while they can handle large volumes of data, they do so as the result of a multitude of parallel, distinct, and isolated micro-operations.

Interactive software

There is, however, another category of software. We could call it 'interactive software'. As the name implies, this includes everything with a user interface, but can also be a long-running batch job, or, as you've already seen, time-sensitive software.

For such software, the impure/pure/impure sandwich architecture is no longer possible. Just think of a UI-based program, like an email client. You compose and send an email, receive a response, then compose a reply, and so on. Every send and receive is impure, as is all the user interface rendering. What happens next depends on what happened before, and everything that happens in the real world is impure.

Have we finally identified the limitations of functional programming?

Hardly. In this series of articles, I'm going to show you how to model pure interactions:

You can skip the Haskell article if you only want to read the F# articles.

This series of articles gives you a comprehensive walkthrough of pure interactions and free monads in F#. For a motivating example, see Pure times, which presents a more realistic example that, on the other hand, doesn't go to the same level of detail.

Summary

The solution to the problem of continuous impure interactions is to model them as a instructions in a (domain-specific) Abstract Syntax Tree (AST), and then using an impure interpreter for the pure AST. You can model the AST as a (free) monad in order to make the required syntax nice.

Next: Hello, pure command-line interaction.


Pure times in F#

Tuesday, 04 July 2017 07:07:00 UTC

A Polling Consumer implementation written in F#.

Previously, you saw how to implement a Polling Consumer in Haskell. This proves that it's possible to write pure functional code modelling long-running interactions with the (impure) world. In this article, you'll see how to port the Haskell code to F#.

For reference, I'll repeat the state transition diagram here:

Polling Consumer state machine transition diagram

For a complete description of the goals and constraints of this particular Polling Consumer implementation, see my earlier Type Driven Development article, or, even better, watch my Pluralsight course Type-Driven Development with F#.

State data types

The program has to keep track of various durations. You can model these as naked TimeSpan values, but in order to add extra type safety, you can, instead, define them as separate types:

type PollDuration = PollDuration of TimeSpan
type IdleDuration = IdleDuration of TimeSpan
type HandleDuration = HandleDuration of TimeSpan
type CycleDuration = {
    PollDuration : PollDuration
    HandleDuration : HandleDuration }

This is a straightforward port of the Haskell code. See the previous article for more details about the motivation for doing this.

You can now define the states of the finite state machine:

type State<'msg> =
| ReadyState of CycleDuration listReceivedMessageState of (CycleDuration list * PollDuration * 'msg)
| NoMessageState of (CycleDuration list * PollDuration)
| StoppedState of CycleDuration list

Again, this is a straight port of the Haskell code.

From instruction set to syntactic sugar

The Polling Consumer must interact with its environment in various ways:

  1. Query the system clock
  2. Poll for messages
  3. Handle messages
  4. Idle
You can model these four cases of interactions as a single discriminated union that describe a small instruction set:

type PollingInstruction<'msg, 'next> =
| CurrentTime of (DateTimeOffset -> 'next)
| Poll of (('msg option * PollDuration-> 'next)
| Handle of ('msg * (HandleDuration -> 'next))
| Idle of (IdleDuration * (IdleDuration -> 'next))

Once more, this is a direct translation of the Haskell code, but from here, this is where your F# code will have to deviate from Haskell. In Haskell, you can, with a single line of code, declare that such a type is a functor. This isn't possible in F#. Instead, you have to explicitly write a map function. This isn't difficult, though. There's a reason that the Haskell compiler can automate this:

// ('a -> 'b) -> PollingInstruction<'c,'a> -> PollingInstruction<'c,'b>
let private mapI f = function
    | CurrentTime next -> CurrentTime (next >> f)
    | Poll next -> Poll (next >> f)
    | Handle (x, next-> Handle (x, next >> f)
    | Idle (x, next-> Idle (x, next >> f)

The function is named mapI, where the I stands for instruction. It's private because the next step is to package the functor in a monad. From that monad, you can define a new functor, so in order to prevent any confusion, I decided to hide the underlying functor from any consumers of the API.

Defining a map function for a generic type like PollingInstruction<'msg, 'next> is well-defined. Pattern-match each union case and return the same case, but with the next function composed with the input function argument f: next >> f. In later articles, you'll see more examples, and you'll see how this recipe is entirely repeatable and automatable.

While a functor isn't an explicit concept in F#, this is how PollingInstruction msg next is a Functor in Haskell. Given a functor, you can produce a free monad. The reason you'd want to do this is that once you have a monad, you can get syntactic sugar. Currently, PollingInstruction<'msg, 'next> only enables you to create Abstract Syntax Trees (ASTs), but the programming experience would be cumbersome and alien. Monads give you automatic do notation in Haskell; in F#, it enables you to write a computation expression builder.

Haskell's type system enables you to make a monad from a functor with a one-liner: type PollingProgram msg = Free (PollingInstruction msg). In F#, you'll have to write some boilerplate code. First, you have to define the monadic type:

type PollingProgram<'msg, 'next> =
| Free of PollingInstruction<'msg, PollingProgram<'msg, 'next>>
| Pure of 'next

You already saw a hint of such a type in the previous article. The PollingProgram<'msg, 'next> discriminated union defines two cases: Free and Pure. The Free case is a PollingInstruction that produces a new PollingProgram as its next step. In essence, this enables you to build an AST, but you also need a signal to stop and return a value from the AST. That's the purpose of the Pure case.

Such a type is only a monad if it defines a bind function (that obey the monad laws):

// ('a -> PollingProgram<'b,'c>) -> PollingProgram<'b,'a>
// -> PollingProgram<'b,'c>
let rec bind f = function
    | Free instruction -> instruction |> mapI (bind f) |> Free
    | Pure x -> f x

This bind function pattern-matches on Free and Pure, respectively. In the Pure case, it simply uses the underlying result value x as an input argument to f. In the Free case, it composes the underlying functor (mapI) with itself recursively. If you find this step obscure, I will not blame you. Just like the implementation of mapI is a bit of boilerplate code, then so is this. It always seems to work this way. If you want to dig deeper into the inner workings of this, then Scott Wlaschin has a detailed explanation.

With the addition of bind PollingProgram<'msg, 'next> becomes a monad (I'm not going to show that the monad laws hold, but they do). Making it a functor is trivial:

//  ('a -> 'b) -> PollingProgram<'c,'a> -> PollingProgram<'c,'b>
let map f = bind (f >> Pure)

The underlying PollingInstruction type was already a functor. This function makes PollingProgram a functor as well.

It'll be convenient with some functions that lifts each PollingInstruction case to a corresponding PollingProgram value. In Haskell, you can use the liftF function for this, but in F# you'll have to be slightly more explicit:

// PollingProgram<'a,DateTimeOffset>
let currentTime = Free (CurrentTime Pure)
 
// PollingProgram<'a,('a option * PollDuration)>
let poll = Free (Poll Pure)
 
// 'a -> PollingProgram<'a,HandleDuration>
let handle msg = Free (Handle (msg, Pure))
 
// IdleDuration -> PollingProgram<'a,IdleDuration>
let idle duration = Free (Idle (duration, Pure))

currentTime and poll aren't even functions, but values. They are, however, small PollingProgram values, so while they look like values (as contrasted to functions), they represent singular executable instructions.

handle and idle are both functions that return PollingProgram values.

You can now implement a small computation expression builder:

type PollingBuilder () =
    member this.Bind (x, f) = Polling.bind f x
    member this.Return x = Pure x
    member this.ReturnFrom x = x
    member this.Zero () = this.Return ()

As you can tell, not much is going on here. The Bind method simply delegates to the above bind function, and the rest are trivial one-liners.

You can create an instance of the PollingBuilder class so that you can write PollingPrograms with syntactic sugar:

let polling = PollingBuilder ()

This enables you to write polling computation expressions. You'll see examples of this shortly.

Most of the code you've seen here is automated in Haskell. This means that while you'll have to explicitly write it in F#, it follows a recipe. Once you get the hang of it, it doesn't take much time. The maintenance overhead of the code is also minimal, because you're essentially implementing a universal abstraction. It's not going to change.

Support functions

Continuing the port of the previous article's Haskell code, you can write a pair of support functions. These are small PollingProgram values:

// IdleDuration -> DateTimeOffset -> PollingProgram<'a,bool>
let private shouldIdle (IdleDuration d) stopBefore = polling {
    let! now = Polling.currentTime
    return now + d < stopBefore }

This shouldIdle function uses the polling computation expression defined above. It first uses the above Polling.currentTime value to get the current time. While Polling.currentTime is a value of the type PollingProgram<'b,DateTimeOffset>, the let! binding makes now a simple DateTimeOffset value. Computation expressions give you the same sort of syntactic sugar that do notation does in Haskell.

If you add now to d, you get a new DateTimeOffset value that represents the time that the program will resume, if it decides to suspend itself for the idle duration. If this time is before stopBefore, the return value is true; otherwise, it's false. Similar to the Haskell example, the return value of shouldIdle isn't just bool, but rather PollingProgram<'a,bool>, because it all takes place inside the polling computation expression.

The function looks impure, but it is pure.

In the same vein, you can implement a shouldPoll function:

// CycleDuration -> TimeSpan
let toTotalCycleTimeSpan x =
    let (PollDuration pd) = x.PollDuration
    let (HandleDuration hd) = x.HandleDuration
    pd + hd
 
// TimeSpan -> DateTimeOffset -> CycleDuration list -> PollingProgram<'a,bool>
let private shouldPoll estimatedDuration stopBefore statistics = polling {
    let expectedHandleDuration =
        statistics
        |> List.map toTotalCycleTimeSpan
        |> Statistics.calculateExpectedDuration estimatedDuration
    let! now = Polling.currentTime
    return now + expectedHandleDuration < stopBefore }

This function uses two helper functions: toTotalCycleTimeSpan and Statistics.calculateExpectedDuration. I've included toTotalCycleTimeSpan in the code shown here, while I'm skipping Statistics.calculateExpectedDuration, because it hasn't changed since the code I show in my Pluralsight course. You can also see the function in the GitHub repository accompanying this article.

Compared to shouldIdle, the shouldPoll function needs an extra (pure) step in order to figure out the expectedHandleDuration, but from there, the two functions are similar.

Transitions

All building blocks are now ready for the finite state machine. In order to break the problem into manageable pieces, you can write a function for each state. Such a function should take as input the data associated with a particular state, and return the next state, based on the input.

The simplest transition is when the program reaches the end state, because there's no way out of that state:

// CycleDuration list -> PollingProgram<'a,State<'b>>
let transitionFromStopped s = polling { return StoppedState s }

The data contained in a StoppedState case has the type CycleDuration list, so the transitionFromStopped function simply lifts such a list to a PollingProgram value by returning a StoppedState value from within a polling computation expression.

Slightly more complex, but still simple, is the transition out of the received state. There's no branching logic involved. You just have to handle the message, measure how much time it takes, append the measurements to previous statistics, and return to the ready state:

// CycleDuration list * PollDuration * 'a -> PollingProgram<'a,State<'b>>
let transitionFromReceived (statistics, pd, msg) = polling {
    let! hd = Polling.handle msg
    return
        { PollDuration = pd; HandleDuration = hd } :: statistics
        |> ReadyState }

This function uses the Polling.handle convenience function to handle the input message. Although the handle function returns a PollingProgram<'a,HandleDuration> value, the let! binding inside of a polling computation expression makes hd a HandleDuration value.

The data contained within a ReceivedMessageState case is a CycleDuration list * PollDuration * 'msg tuple. That's the input argument to the transitionFromReceived function, which immediately pattern-matches the tuple's three elements into statistics, pd, and msg.

The pd element is the PollDuration - i.e. the time it took to reach the received state. The hd value returned by Polling.handle gives you the time it took to handle the message. From those two values you can create a new CycleDuration value, and cons (::) it onto the previous statistics. This returns an updated list of statistics that you can pipe to the ReadyState case constructor.

ReadyState in itself creates a new State<'msg> value, but since all of this takes place inside a polling computation expression, the return type of the function becomes PollingProgram<'a,State<'b>>.

The transitionFromReceived function handles the state when the program has received a message, but you also need to handle the state when no message was received:

// IdleDuration -> DateTimeOffset -> CycleDuration list * 'a
// -> PollingProgram<'b,State<'c>>
let transitionFromNoMessage d stopBefore (statistics, _) = polling {
    let! b = shouldIdle d stopBefore
    if b then
        do! Polling.idle d |> Polling.map ignore
        return ReadyState statistics
    else return StoppedState statistics }

This function first calls the shouldIdle support function. Similar to Haskell, you can see how you can compose larger PollingPrograms from smaller PollingProgram values - just like you can compose 'normal' functions from smaller functions.

With the syntactic sugar in place, b is simply a bool value that you can use in a standard if/then/else expression. If b is false, then return a StoppedState value; otherwise, continue with the next steps.

Polling.idle returns the duration of the suspension, but you don't actually need this data, so you can ignore it. When Polling.idle returns, you can return a ReadyState value.

It may look as though that do! expression is a blocking call, but it really isn't. The transitionFromNoMessage function only builds an Abstract Syntax Tree, where one of the instructions suggests that an interpreter could block. Unless evaluated by an impure interpreter, transitionFromNoMessage is pure.

The final transition is the most complex, because there are three possible outcomes:

// TimeSpan -> DateTimeOffset -> CycleDuration list
// -> PollingProgram<'a,State<'a>>
let transitionFromReady estimatedDuration stopBefore statistics = polling {
    let! b = shouldPoll estimatedDuration stopBefore statistics
    if b then
        let! pollResult = Polling.poll
        match pollResult with
        | Some msg, pd -> return ReceivedMessageState (statistics, pd, msg)
        | None, pd -> return NoMessageState (statistics, pd)
    else return StoppedState statistics }

In the same way that transitionFromNoMessage uses shouldIdle, the transitionFromReady function uses the shouldPoll support function to decide whether or not to keep going. If b is false, it returns a StoppedState value.

Otherwise, it goes on to poll. Thanks to all the syntactic sugar, pollResult is an 'a option * PollDuration value. As always, when you have a discriminated union, you can handle all cases with pattern matching (and the compiler will help you keep track of whether or not you've handled all of them).

In the Some case, you have a message, and the duration it took to poll for that message. This is all the data you need to return a ReceivedMessageState value.

In the None case, you also have the poll duration pd; return a NoMessageState value.

That's four transition functions that you can combine in a single function that, for any state, returns a new state:

// TimeSpan -> IdleDuration -> DateTimeOffset -> State<'a>
// -> PollingProgram<'a,State<'a>>
let transition estimatedDuration idleDuration stopBefore = function
    | ReadyState s -> transitionFromReady estimatedDuration stopBefore s
    | ReceivedMessageState s -> transitionFromReceived s
    | NoMessageState s -> transitionFromNoMessage idleDuration stopBefore s
    | StoppedState s -> transitionFromStopped s

You simply pattern-match the (implicit) input argument with the four state cases, and call the appropriate transition function for each case.

Interpretation

The transition function is pure. It returns a PollingProgram value. How do you turn it into something that performs real work?

You write an interpreter:

// PollingProgram<Msg,'a> -> 'a
let rec interpret = function
    | Pure x -> x
    | Free (CurrentTime next)   -> DateTimeOffset.Now |> next |> interpret
    | Free (Poll next)          -> Imp.poll ()        |> next |> interpret
    | Free (Handle (msg, next)) -> Imp.handle msg     |> next |> interpret
    | Free (Idle (d, next))     -> Imp.idle d         |> next |> interpret

A PollingProgram is either a Pure or a Free case. In the Free case, the contained data is a PollingInstruction value, which can be one of four separate cases. With pattern matching, the interpreter handles all five cases.

In the Pure case, it returns the value, but in all the Free cases, it recursively calls itself after having first followed the instruction in each PollingInstruction case. For instance, when the instruction is CurrentTime, it invokes DateTimeOffset.Now, passes the return value (a DateTimeOffset value) to the next continuation, and then recursively calls interpret. The next instruction, then, could be another Free case, or it could be Pure.

The other three instruction cases delegate to implementation functions defined in an Imp module. I'm not going to show them here. They're normal, although impure, F# functions.

Execution

You're almost done. You have a function that returns a new state for any given input state, as well as an interpreter. You need a function that can repeat this in a loop until it reaches StoppedState:

// TimeSpan -> IdleDuration -> DateTimeOffset -> State<Msg> -> State<Msg>
let rec run estimatedDuration idleDuration stopBefore s =
    let ns =
        PollingConsumer.transition estimatedDuration idleDuration stopBefore s
        |> interpret
    match ns with
    | PollingConsumer.StoppedState _ -> ns
    | _ -> run estimatedDuration idleDuration stopBefore ns

This function calls PollingConsumer.transition with the input state s, which returns a new PollingProgram<Msg,PollingConsumer.State<Msg>> value that you can pipe to the interpret function. That gives you the new state ns. If ns is a StoppedState, you return; otherwise, you recurse into run for another round.

Finally, you can write the entry point for the application:

[<EntryPoint>]
let main _ =
    let timeAtEntry = DateTimeOffset.Now
 
    printOnEntry timeAtEntry
 
    let stopBefore = timeAtEntry + limit
    let estimatedDuration = TimeSpan.FromSeconds 2.
    let idleDuration = TimeSpan.FromSeconds 5. |> IdleDuration
 
    let durations =
        PollingConsumer.ReadyState []
        |> run estimatedDuration idleDuration stopBefore
        |> PollingConsumer.durations
        |> List.map PollingConsumer.toTotalCycleTimeSpan
    
    printOnExit timeAtEntry durations
 
    // Return 0. This indicates success.
    0

This defines an estimated duration of 2 seconds, an idle duration of 5 seconds, and a maximum run time of 60 seconds (limit). The initial state is ReadyState with no prior statistics. Pass all these arguments to the run function, and you have a running program.

This function also uses a few printout functions that I'm not going to show here. When you run the program, you should see output like this:

Started polling at 11:18:28.

Polling
Handling
Polling
Handling
Polling
Sleeping
Polling
Sleeping
Polling
Handling
Polling
Handling
Polling
Sleeping
Polling
Sleeping
Polling
Sleeping
Polling
Handling
Polling
Sleeping
Polling
Sleeping
Polling
Sleeping
Polling
Sleeping
Polling
Handling

Stopped polling at 11:19:26.
Elapsed time: 00:00:58.4428980.
Handled 6 message(s).
Average duration: 00:00:01.0550346
Standard deviation: 00:00:00.3970599

It does, indeed, exit before 60 seconds have elapsed.

Summary

You can model long-running interactions with an Abstract Syntax Tree. Without computation expressions, writing programs as 'raw' ASTs would be cumbersome, but turning the AST into a (free) monad makes it all quite palatable.

Haskell code with a free monad can be ported to F#, although some boilerplate code is required. That code, however, is unlikely to be much of a burden, because it follows a well-known recipe that implements a universal abstraction.

For more details on how to write free monads in F#, see Pure interactions.


Page 1 of 37

"Our team wholeheartedly endorses Mark. His expert service provides tremendous value."
Hire me!