ploeh blog danish software design

A port of the corresponding F# code.

This article is an instalment in a short series of articles on the Range kata. In the previous article I made a pass at the kata in F#, using property-based testing with Hedgehog to generate test data.

In the conclusion I mused about the properties I was able to come up with. Is it possible to describe open, closed, and mixed ranges in a way that's less coupled to the implementation? To be honest, I still don't have an answer to that question. Instead, in this article, I describe a straight port of the F# code to C#. There's value in that, too, for people who wonder how to reap the benefits of F# in C#.

The code is available on GitHub.

First property #

Both F# and C# are .NET languages. They run in the same substrate, and are interoperable. While Hedgehog is written in F#, it's possible to consume F# libraries from C#, and vice versa. I've done this multiple times with FsCheck, but I admit to never having tried it with Hedgehog.

If you want to try property-based testing in C#, a third alternative is available: CsCheck. It's written in C# and is more idiomatic in that context. While I sometimes still use FsCheck from C#, I often choose CsCheck for didactic reasons.

The first property I wrote was a direct port of the idea of the first property I wrote in F#:

[Fact]
public void ClosedRangeContainsList()
{
    (from xs in Gen.Short.Enumerable.Nonempty
     let min = xs.Min()
     let max = xs.Max()
     select (xs, min, max))
    .Sample(t =>
    {
        var sut = new Range<short>(
            new ClosedEndpoint<short>(t.min),
            new ClosedEndpoint<short>(t.max));
 
        var actual = sut.Contains(t.xs);
 
        Assert.True(actual, $"Expected {t.xs} to be contained in {sut}.");
    });
}

This test (or property, if you will) uses a technique that I often use with property-based testing. I'm still searching for a catchy name for this, but here we may call it something like reverse test-case assembly. My goal is to test a predicate, and this particular property should verify that for a given Equivalence Class, the predicate is always true.

While we may think of an Equivalence Class as a set from which we pick test cases, I don't actually have a full enumeration of such a set. I can't have that, since that set is infinitely big. Instead of randomly picking values from a set that I can't fully populate, I instead carefully pick test case values in such a way that they would all belong to the same set partition (Equivalence Class).

The test name suggests the test case: I'd like to verify that given I have a closed range, when I ask it whether a list within that range is contained, then the answer is true. How do I pick such a test case?

I do it in reverse. You can say that the sampling is the dual of the test. I start with a list (xs) and only then do I create a range that contains it. Since the first test case is for a closed range, the min and max values are sufficient to define such a range.

How do I pass that property?

Degenerately, as is often the case with TDD beginnings:

public bool Contains(IEnumerable<T> candidates)
{
    return true;
}

Even though the ClosedRangeContainsList property effectively executes a hundred test cases, the Devil's Advocate can easily ignore that and instead return hard-coded true.

Endpoint sum type #

I'm not going to bore you with the remaining properties. The repository is available on GitHub if you're interested in those details.

If you've programmed in F# for some time, you typically miss algebraic data types when forced to return to C#. A language like C# does have product types, but lack native sum types. Even so, not all is lost. I've previously demonstrated that you can employ the Visitor pattern to encode a sum type. Another option is to use Church encoding, which I've decided to do here.

When choosing between Church encoding and the Visitor pattern, Visitor is more object-oriented (after all, it's an original GoF design pattern), but Church encoding has fewer moving parts. Since I was just doing an exercise, I went for the simpler implementation.

An Endpoint object should allow one of two cases: Open or Closed. To avoid primitive obsession I gave the class a private constructor:

public sealed class Endpoint<T>
{
    private readonly T value;
    private readonly bool isClosed;
 
    private Endpoint(T value, bool isClosed)
    {
        this.value = value;
        this.isClosed = isClosed;
    }

Since the constructor is private you need another way to create Endpoint objects. Two factory methods provide that affordance:

public static Endpoint<T> Closed<T>(T value)
{
    return Endpoint<T>.Closed(value);
}
 
public static Endpoint<T> Open<T>(T value)
{
    return Endpoint<T>.Open(value);
}

The heart of the Church encoding is the Match method:

public TResult Match<TResult>(
    Func<T, TResult> whenClosed,
    Func<T, TResult> whenOpen)
{
    if (isClosed)
        return whenClosed(value);
    else
        return whenOpen(value);
}

Such an API is an example of poka-yoke because it obliges you to deal with both cases. The compiler will keep you honest: Did you remember to deal with both the open and the closed case? When calling the Match method, you must supply both arguments, or your code doesn't compile. Make illegal states unrepresentable.

Containment #

With the Endpoint class in place, you can implement a Range class.

public sealed class Range<T> where T : IComparable<T>

It made sense to me to constrain the T type argument to IComparable<T>, although it's possible that I could have deferred that constraint to the actual Contains method, like I did with my Haskell implementation.

A Range holds two Endpoint values:

public Range(Endpoint<T> min, Endpoint<T> max)
{
    this.min = min;
    this.max = max;
}

The Contains method makes use of the built-in All method, using a private helper function as the predicate:

private bool IsInRange(T candidate)
{
    return min.Match(
        whenClosed: l => max.Match(
            whenClosed: h => l.CompareTo(candidate) <= 0 && candidate.CompareTo(h) <= 0,
            whenOpen:   h => l.CompareTo(candidate) <= 0 && candidate.CompareTo(h) <  0),
        whenOpen: l => max.Match(
            whenClosed: h => l.CompareTo(candidate) <  0 && candidate.CompareTo(h) <= 0,
            whenOpen:   h => l.CompareTo(candidate) <  0 && candidate.CompareTo(h) <  0));
}

This implementation performs a nested Match to arrive at the appropriate answer. The code isn't as elegant or readable as its F# counterpart, but it comes with comparable compile-time safety. You can't forget a combination, because if you do, your code isn't going to compile.

Still, you can't deny that C# involves more ceremony.

Conclusion #

Once you know how, it's not that difficult to port a functional design from F# or Haskell to a language like C#. The resulting code tends to be more complicated, but to a large degree, it's possible to retain the type safety.

In this article you saw a sketch of how to make that transition, using the Range kata as an example. The resulting C# API is perfectly serviceable, as the test code demonstrates.

Now that we have covered the fundamentals of the Range kata we have learned enough about it to go beyond the exercise and examine some more abstract properties.

Next: Range as a functor.

This time with some property-based testing.

This article is an instalment in a short series of articles on the Range kata. In the previous article I described my first attempt at the kata, and also complained that I had to think of test cases myself. When I find it tedious coming up with new test cases, I usually start to wonder if it'd be easier to use property-based testing.

Thus, when I decided to revisit the kata, the variation that I was most interested in pursuing was to explore whether it would make sense to use property-based testing instead of a set of existing examples.

Since I also wanted to do the second attempt in F#, I had a choice between FsCheck and Hedgehog. Each have their strengths and weaknesses, but since I already know FsCheck so well, I decided to go with Hedgehog.

I also soon discovered that I had no interest in developing the full suite of capabilities implied by the kata. Instead, I decided to focus on just the data structure itself, as well as the contains function. As in the previous article, this function can also be used to cover the kata's ContainsRange feature.

Getting started #

There's no rule that you can't combine property-based testing with test-driven development (TDD). On the contrary, that's how I often do it. In this exercise, I first wrote this test:

[<Fact>]
let ``Closed range contains list`` () = Property.check <| property {
    let! xs = Gen.int16 (Range.linearBounded ()) |> Gen.list (Range.linear 1 99)
    let min = List.min xs
    let max = List.max xs
 
    let actual = (Closed min, Closed max) |> Range.contains xs
 
    Assert.True (actual, sprintf "Range [%i, %i] expected to contain list." min max) }

We have to be careful when reading and understanding this code: There are two Range modules in action here!

Hedgehog comes with a Range module that you must use to define how it samples values from domains. Examples of that here are Range.linearBounded and Range.linear.

On the other hand, I've defined my contains function in a Range module, too. As long as there's no ambiguity, the F# compiler doesn't have a problem with that. Since there's no contains function in the Hedgehog Range module, the F# compiler isn't confused.

We humans, on the other hand, might be confused, and had this been a code base that I had to maintain for years, I might seriously consider whether I should rename my own Range module to something else, like Interval, perhaps.

In any case, the first test (or property, if you will) uses a technique that I often use with property-based testing. I'm still searching for a catchy name for this, but here we may call it something like reverse test-case assembly. My goal is to test a predicate, and this particular property should verify that for a given Equivalence Class, the predicate is always true.

While we may think of an Equivalence Class as a set from which we pick test cases, I don't actually have a full enumeration of such a set. I can't have that, since that set is infinitely big. Instead of randomly picking values from a set that I can't fully populate, I instead carefully pick test case values in such a way that they would all belong to the same set partition (Equivalence Class).

The test name suggests the test case: I'd like to verify that given I have a closed range, when I ask it whether a list within that range is contained, then the answer is true. How do I pick such a test case?

I do it in reverse. You can say that the sampling is the dual of the test. I start with a list (xs) and only then do I create a range that contains it. Since the first test case is for a closed range, the min and max values are sufficient to define such a range.

How do I pass that property?

Degenerately, as is often the case with TDD beginnings:

module Range =
    let contains _ _ = true

Even though the Closed range contains list property effectively executes a hundred test cases, the Devil's Advocate can easily ignore that and instead return hard-coded true.

More properties are required to flesh out the behaviour of the function.

Open range #

While I do keep the transformation priority premise in mind when picking the next test (or, here, property), I'm rarely particularly analytic about it. Since the first property tests that a closed range barely contains a list of values from its minimum to its maximum, it seemed like a promising next step to consider the case where the range consisted of open endpoints. That was the second test I wrote, then:

[<Fact>]
let ``Open range doesn't contain endpoints`` () = Property.check <| property {
    let! min = Gen.int32 (Range.linearBounded ())
    let! max = Gen.int32 (Range.linearBounded ())
 
    let actual = (Open min, Open max) |> Range.contains [min; max]
 
    Assert.False (actual, sprintf "Range (%i, %i) expected not to contain list." min max) }

This property simply states that if you query the contains predicate about a list that only contains the endpoints of an open range, then the answer is false because the endpoints are Open.

One implementation that passes both tests is this one:

module Range =
    let contains _ endpoints =
        match endpoints with
        | Open _, Open _ -> false
        | _ -> true

This implementation is obviously still incorrect, but we have reason to believe that we're moving closer to something that will eventually work.

Tick-tock #

In the spirit of the transformation priority premise, I've often found that when test-driving a predicate, I seem to fall into a tick-tock pattern where I alternate between tests for a true return value, followed by a test for a false return value, or the other way around. This was also the case here. The previous test was for a false value, so the third test requires true to be returned:

[<Fact>]
let ``Open range contains list`` () = Property.check <| property {
    let! xs = Gen.int64 (Range.linearBounded ()) |> Gen.list (Range.linear 1 99)
    let min = List.min xs - 1L
    let max = List.max xs + 1L
 
    let actual = (Open min, Open max) |> Range.contains xs
 
    Assert.True (actual, sprintf "Range (%i, %i) expected to contain list." min max) }

This then led to this implementation of the contains function:

module Range =
    let contains ys endpoints =
        match endpoints with
        | Open x, Open z ->
            ys |> List.forall (fun y -> x < y && y < z)
        | _ -> true

Following up on the above true-demanding test, I added one that tested a false scenario:

[<Fact>]
let ``Open-closed range doesn't contain endpoints`` () = Property.check <| property {
    let! min = Gen.int16 (Range.linearBounded ())
    let! max = Gen.int16 (Range.linearBounded ())
 
    let actual = (Open min, Closed max) |> Range.contains [min; max]
 
    Assert.False (actual, sprintf "Range (%i, %i] expected not to contain list." min max) }

This again led to this implementation:

module Range =
    let contains ys endpoints =
        match endpoints with
        | Open x, Open z ->
            ys |> List.forall (fun y -> x < y && y < z)
        | Open x, Closed z -> false
        | _ -> true

I had to add four more tests before I felt confident that I had the right implementation. I'm not going to show them all here, but you can look at the repository on GitHub if you're interested in the interim steps.

Types and functionality #

So far I had treated a range as a pair (two-tuple), just as I had done with the code in my first attempt. I did, however, have a few other things planned for this code base, so I introduced a set of explicit types:

type Endpoint<'a> = Open of 'a | Closed of 'a
 
type Range<'a> = { LowerBound : Endpoint<'a>; UpperBound : Endpoint<'a> }

The Range record type is isomorphic to a pair of Endpoint values, so it's not strictly required, but does make things more explicit.

To support the new type, I added an ofEndpoints function, and finalized the implementation of contains:

module Range =
    let ofEndpoints (lowerBound, upperBound) =
        { LowerBound = lowerBound; UpperBound = upperBound }
 
    let contains ys r =
        match r.LowerBound, r.UpperBound with
        |   Open x,   Open z -> ys |> List.forall (fun y -> x  < y && y  < z)
        |   Open x, Closed z -> ys |> List.forall (fun y -> x  < y && y <= z)
        | Closed x,   Open z -> ys |> List.forall (fun y -> x <= y && y  < z)
        | Closed x, Closed z -> ys |> List.forall (fun y -> x <= y && y <= z)

As is so often the case in F#, pattern matching makes such functions a pleasure to implement.

Conclusion #

I was curious whether using property-based testing would make the development process of the Range kata simpler. While each property was simple, I still had to write eight of them before I felt I'd fully described the problem. This doesn't seem like much of an improvement over the example-driven approach I took the first time around. It seems to be a comparable amount of code, and on one hand a property is more abstract than an example, but on the hand usually also covers more ground. I feel more confident that this implementation works, because I know that it's being exercised more rigorously.

When I find myself writing a property per branch, so to speak, I always feel that I missed a better way to describe the problem. As an example, for years I would demonstrate how to test the FizzBuzz kata with property-based testing by dividing the problem into Equivalence Classes and then writing a property for each partition. Just as I've done here. This is usually possible, but smells of being too coupled to the implementation.

Sometimes, if you think about the problem long enough, you may be able to produce an alternative set of properties that describe the problem in a way that's entirely decoupled from the implementation. After years, I finally managed to do that with the FizzBuzz kata.

I didn't succeed doing that with the Range kata this time around, but maybe later.

Next: A Range kata implementation in C#.

A first crack at the exercise.

This article is an instalment in a short series of articles on the Range kata. Here I describe my first attempt at the exercise. As I usually advise people on doing katas, the first time you try your hand at a kata, use the language with which you're most comfortable. To be honest, I may be most habituated to C#, having programmed in it since 2002, but on the other hand, I currently 'think in Haskell', and am often frustrated with C#'s lack of structural equality, higher-order abstractions, and support for functional expressions.

Thus, I usually start with Haskell even though I always find myself struggling with the ecosystem. If you do, too, the source code is available on GitHub.

I took my own advice by setting out with the explicit intent to follow the Range kata description as closely as possible. This kata doesn't beat about the bush, but instead just dumps a set of test cases on you. It wasn't clear if this is the most useful set of tests, or whether the order in which they're represented is the one most conducive to a good experience of test-driven development, but there was only one way to find out.

I quickly learned, however, that the suggested test cases were insufficient in describing the behaviour in enough details.

Containment #

I started by adding the first two test cases as inlined HUnit test lists:

"integer range contains" ~: do
  (r, candidate, expected) <-
    [
      ((Closed 2, Open 6), [2,4], True),
      ((Closed 2, Open 6), [-1,1,6,10], False)
    ]
  let actual = r `contains` candidate
  return $ expected ~=? actual

I wasn't particularly keen on going full Devil's Advocate on the exercise. I could, on the other hand, trivially pass both tests with this obviously degenerate implementation:

import Data.List
 
data Endpoint a = Open a | Closed a deriving (Eq, Show)
 
contains _ candidate = [2] `isPrefixOf` candidate

Reluctantly, I had to invent some additional test cases:

"integer range contains" ~: do
  (r, candidate, expected) <-
    [
      ((Closed   2 ,   Open  6),       [2,4],  True),
      ((Closed   2 ,   Open  6), [-1,1,6,10], False),
      ((Closed (-1), Closed 10), [-1,1,6,10],  True),
      ((Closed (-1),   Open 10), [-1,1,6,10], False),
      ((Closed (-1),   Open 10),  [-1,1,6,9],  True),
      ((  Open   2,  Closed  6),     [3,5,6],  True),
      ((  Open   2,    Open  6),       [2,5], False),
      ((  Open   2,    Open  6),          [],  True),
      ((Closed   2,  Closed  6),     [3,7,4], False)
    ]
  let actual = r `contains` candidate
  return $ expected ~=? actual

This was when I began to wonder whether it would have been easier to use property-based testing. That would entail, however, a departure from the kata's suggested test cases, so I decided to stick to the plan and then perhaps return to property-based testing when repeating the exercise.

Ultimately I implemented the contains function this way:

contains :: (Foldable t, Ord a) => (Endpoint a, Endpoint a) -> t a -> Bool
contains (lowerBound, upperBound) =
  let isHighEnough = case lowerBound of
        Closed x -> (x <=)
        Open   x -> (x <)
      isLowEnough = case upperBound of
        Closed y -> (<= y)
        Open   y ->  (< y)
      isContained x = isHighEnough x && isLowEnough x
  in all isContained

In some ways it seems a bit verbose to me, but I couldn't easily think of a simpler implementation.

One of the features I find so fascinating about Haskell is how general it enables me to be. While the tests use integers for concision, the contains function works with any Ord instance; not only Integer, but also Double, Word, Day, TimeOfDay, or some new type I can't even predict.

All points #

The next function suggested by the kata is a function to enumerate all points in a range. There's only a single test case, so again I added some more:

"getAllPoints" ~: do
  (r, expected) <-
    [
      ((Closed 2,   Open 6), [2..5]),
      ((Closed 4,   Open 8), [4..7]),
      ((Closed 2, Closed 6), [2..6]),
      ((Closed 4, Closed 8), [4..8]),
      ((  Open 2, Closed 6), [3..6]),
      ((  Open 4, Closed 8), [5..8]),
      ((  Open 2,   Open 6), [3..5]),
      ((  Open 4,   Open 8), [5..7])
    ]
  let actual = allPoints r
  return $ expected ~=? actual

Ultimately, after I'd implemented the next feature, I refactored the allPoints function to make use of it, and it became a simple one-liner:

allPoints :: (Enum a, Num a) => (Endpoint a, Endpoint a) -> [a]
allPoints = uncurry enumFromTo . endpoints

The allPoints function also enabled me to express the kata's ContainsRange test cases without introducing a new API:

"ContainsRange" ~: do
  (r, candidate, expected) <-
    [
      ((Closed 2,   Open  5), allPoints (Closed 7, Open   10), False),
      ((Closed 2,   Open  5), allPoints (Closed 3, Open   10), False),
      ((Closed 3,   Open  5), allPoints (Closed 2, Open   10), False),
      ((Closed 2,   Open 10), allPoints (Closed 3, Closed  5),  True),
      ((Closed 3, Closed  5), allPoints (Closed 3, Open    5),  True)
    ]
  let actual = r `contains` candidate
  return $ expected ~=? actual

As I've already mentioned, the above implementation of allPoints is based on the next feature, endpoints.

Endpoints #

The kata also suggests a function to return the two endpoints of a range, as well as some test cases to describe it. Once more, I had to add more test cases to adequately describe the desired functionality:

"endPoints" ~: do
  (r, expected) <-
    [
      ((Closed 2,   Open 6), (2, 5)),
      ((Closed 1,   Open 7), (1, 6)),
      ((Closed 2, Closed 6), (2, 6)),
      ((Closed 1, Closed 7), (1, 7)),
      ((  Open 2,   Open 6), (3, 5)),
      ((  Open 1,   Open 7), (2, 6)),
      ((  Open 2, Closed 6), (3, 6)),
      ((  Open 1, Closed 7), (2, 7))
    ]
  let actual = endpoints r
  return $ expected ~=? actual

The implementation is fairly trivial:

endpoints :: (Num a1, Num a2) => (Endpoint a2, Endpoint a1) -> (a2, a1)
endpoints (Closed x, Closed y) = (x  , y)
endpoints (Closed x,   Open y) = (x  , y-1)
endpoints (  Open x, Closed y) = (x+1, y)
endpoints (  Open x,   Open y) = (x+1, y-1)

One attractive quality of algebraic data types is that the 'algebra' of the type(s) tell you how many cases you need to pattern-match against. Since I'm treating a range as a pair of Endpoint values, and since each Endpoint can be one of two cases (Open or Closed), there's exactly 2 * 2 = 4 possible combinations (since a tuple is a product type).

That fits with the number of pattern-matches required to implement the function.

Overlapping ranges #

The final interesting feature is a predicate to determine whether one range overlaps another. As has become a refrain by now, I didn't find the suggested test cases sufficient to describe the desired behaviour, so I had to add a few more:

"overlapsRange" ~: do
  (r, candidate, expected) <-
    [
      ((Closed 2, Open  5), (Closed 7, Open 10), False),
      ((Closed 2, Open 10), (Closed 3, Open  5),  True),
      ((Closed 3, Open  5), (Closed 3, Open  5),  True),
      ((Closed 2, Open  5), (Closed 3, Open 10),  True),
      ((Closed 3, Open  5), (Closed 2, Open 10),  True),
      ((Closed 3, Open  5), (Closed 1, Open  3), False),
      ((Closed 3, Open  5), (Closed 5, Open  7), False)
    ]
  let actual = r `overlaps` candidate
  return $ expected ~=? actual

I'm not entirely happy with the implementation:

overlaps :: (Ord a1, Ord a2) =>
            (Endpoint a1, Endpoint a2) -> (Endpoint a2, Endpoint a1) -> Bool
overlaps (l1, h1) (l2, h2) =
  let less (Closed x) (Closed y) = x <= y
      less (Closed x)   (Open y) = x <  y
      less   (Open x) (Closed y) = x <  y
      less   (Open x)   (Open y) = x <  y
  in l1 `less` h2 && l2 `less` h1

Noth that the code presented here is problematic in isolation, but if you compare it to the above contains function, there seems to be some repetition going on. Still, it's not quite the same, but the code looks similar enough that it bothers me. I feel that some kind of abstraction is sitting there, right before my nose, mocking me because I can't see it. Still, the code isn't completely duplicated, and even if it was, I can always invoke the rule of three and let it remain as it is.

Which is ultimately what I did.

Equality #

The kata also suggests some test cases to verify that it's possible to compare two ranges for equality. Dutifully I added those test cases to the code base, even though I knew that they'd automatically pass.

"Equals" ~: do
  (x, y, expected) <-
    [
      ((Closed 3, Open  5), (Closed 3, Open  5),  True),
      ((Closed 2, Open 10), (Closed 3, Open  5), False),
      ((Closed 2, Open  5), (Closed 3, Open 10), False),
      ((Closed 3, Open  5), (Closed 2, Open 10), False)
    ]
  let actual = x == y
  return $ expected ~=? actual

In the beginning of this article, I called attention to C#'s regrettable lack of structural equality. Here's an example of what I mean. In Haskell, these tests automatically pass because Endpoint is an Eq instance (by declaration), and all pairs of Eq instances are themselves Eq instances. Simple, elegant, powerful.

Conclusion #

As a first pass at the (admittedly uncomplicated) Range kata, I tried to follow the 'plan' implied by the kata description's test cases. I quickly became frustrated with their lack of completion. They were adequate in indicating to a human (me) what the desired behaviour should be, but insufficient to satisfactorily describe the desired behaviour.

I could, of course, have stuck with only those test cases, and instead of employing the Devil's Advocate technique (which I actively tried to avoid) made an honest effort to implement the functionality.

The things is, however, that I don't trust myself. At its essence, the Range kata is all about edge cases, which are where most bugs tend to lurk. Thus, these are exactly the cases that should be covered by tests.

Having made enough 'dumb' programming mistakes during my career, I didn't trust myself to be able to write correct implementations without more test coverage than originally suggested. That's the reason I added more tests.

On the other hand, I more than once speculated whether property-based testing would make this work easier. I decided to pursue that idea during my second pass at the kata.

Next: A Range kata implementation in F#.

Comments

ghci> 1%2
1 % 2

ghci> succ $ 1%2
3 % 2
ghci> succ $ succ $ 1%2
5 % 2

ghci> succ 1.5
2.5
ghci> succ $ succ 1.5
3.5

In the languages I usually employ.

The Range kata is succinct, bordering on the spartan in both description and requirements. To be honest, it's hardly the most inspiring kata available, and yet it may help showcase a few interesting points about software design in general. It's what it demonstrates about functors that makes it marginally interesting.

In this short article series I first cover a few incarnations of the kata in my usual programming languages, and then conclude by looking at range as a functor.

The article series contains the following articles:

• A Range kata implementation in Haskell
• A Range kata implementation in F#
• A Range kata implementation in C#
• Range as a functor

I didn't take the same approaches through all three exercises. An important point about doing katas is to learn something, and when you've done the kata once, you've already gained some knowledge that can't easily be unlearned. Thus, on the second, or third time through, it's only natural to apply that knowledge, but then try different tactics to solve the problem in a different way. That's what I did here, starting with Haskell, proceeding with F#, and concluding with C#.

Next: A Range kata implementation in Haskell.

Using System.Text.Json, with and without Reflection.

This article is part of a short series of articles about serialization with and without Reflection. In this instalment I'll explore some options for serializing JSON with C# using the API built into .NET: System.Text.Json. I'm not going use Json.NET in this article, but I've done similar things with that library in the past, so what's here is, at least, somewhat generalizable.

Since the API is the same, the only difference from the previous article is the language syntax.

Natural numbers #

Before we start investigating how to serialize to and from JSON, we must have something to serialize. As described in the introductory article we'd like to parse and write restaurant table configurations like this:

{
  "singleTable": {
    "capacity": 16,
    "minimalReservation": 10
  }
}

On the other hand, I'd like to represent the Domain Model in a way that encapsulates the rules governing the model, making illegal states unrepresentable. Even though that's a catchphrase associated with functional programming, it applies equally well to a statically typed object-oriented language like C#.

As the first step, we observe that the numbers involved are all natural numbers. In C# it's rarer to define predicative data types than in a language like F#, but people should do it more.

public readonly struct NaturalNumber : IEquatable<NaturalNumber>
{
    private readonly int value;
 
    public NaturalNumber(int value)
    {
        if (value < 1)
            throw new ArgumentOutOfRangeException(
                nameof(value),
                "Value must be a natural number greater than zero.");
        this.value = value;
    }
 
    public static NaturalNumber? TryCreate(int candidate)
    {
        if (candidate < 1)
            return null;
        return new NaturalNumber(candidate);
    }
 
    public static bool operator <(NaturalNumber left, NaturalNumber right)
    {
        return left.value < right.value;
    }
 
    public static bool operator >(NaturalNumber left, NaturalNumber right)
    {
        return left.value > right.value;
    }
 
    public static bool operator <=(NaturalNumber left, NaturalNumber right)
    {
        return left.value <= right.value;
    }
 
    public static bool operator >=(NaturalNumber left, NaturalNumber right)
    {
        return left.value >= right.value;
    }
 
    public static bool operator ==(NaturalNumber left, NaturalNumber right)
    {
        return left.value == right.value;
    }
 
    public static bool operator !=(NaturalNumber left, NaturalNumber right)
    {
        return left.value != right.value;
    }
 
    public static explicit operator int(NaturalNumber number)
    {
        return number.value;
    }
 
    public override bool Equals(object? obj)
    {
        return obj is NaturalNumber number && Equals(number);
    }
 
    public bool Equals(NaturalNumber other)
    {
        return value == other.value;
    }
 
    public override int GetHashCode()
    {
        return HashCode.Combine(value);
    }
}

When comparing all that boilerplate code to the three lines required to achieve the same result in F#, it seems, at first glance, understandable that C# developers rarely reach for that option. Still, typing is not a programming bottleneck, and most of that code was generated by a combination of Visual Studio and GitHub Copilot.

The TryCreate method may not be strictly necessary, but I consider it good practice to give client code a way to perform a fault-prone operation in a safe manner, without having to resort to a try/catch construct.

That's it for natural numbers. 72 lines of code. Compare that to the F# implementation, which required three lines of code. Syntax does matter.

Domain Model #

Modelling a restaurant table follows in the same vein. One invariant I would like to enforce is that for a 'single' table, the minimal reservation should be a NaturalNumber less than or equal to the table's capacity. It doesn't make sense to configure a table for four with a minimum reservation of six.

In the same spirit as above, then, define this type:

public readonly struct Table
{
    private readonly NaturalNumber capacity;
    private readonly NaturalNumber? minimalReservation;
 
    private Table(NaturalNumber capacity, NaturalNumber? minimalReservation)
    {
        this.capacity = capacity;
        this.minimalReservation = minimalReservation;
    }
 
    public static Table? TryCreateSingle(int capacity, int minimalReservation)
    {
        var cap = NaturalNumber.TryCreate(capacity);
        if (cap is null)
            return null;
        var min = NaturalNumber.TryCreate(minimalReservation);
        if (min is null)
            return null;
 
        if (cap < min)
            return null;
 
        return new Table(cap.Value, min.Value);
    }
 
    public static Table? TryCreateCommunal(int capacity)
    {
        var cap = NaturalNumber.TryCreate(capacity);
        if (cap is null)
            return null;
 
        return new Table(cap.Value, null);
    }
 
    public T Accept<T>(ITableVisitor<T> visitor)
    {
        if (minimalReservation is null)
            return visitor.VisitCommunal(capacity);
        else
            return visitor.VisitSingle(capacity, minimalReservation.Value);
    }
}

Here I've Visitor-encoded the sum type that Table is. It can either be a 'single' table or a communal table.

Notice that TryCreateSingle checks the invariant that the capacity must be greater than or equal to the minimalReservation.

The point of this little exercise, so far, is that it encapsulates the contract implied by the Domain Model. It does this by using the static type system to its advantage.

JSON serialization by hand #

At the boundaries of applications, however, there are no static types. Is the static type system still useful in that situation?

For a long time, the most popular .NET library for JSON serialization was Json.NET, but these days I find the built-in API offered in the System.Text.Json namespace adequate. This is also the case here.

The original rationale for this article series was to demonstrate how serialization can be done without Reflection, so I'll start there and return to Reflection later.

In this article series, I consider the JSON format fixed. A single table should be rendered as shown above, and a communal table should be rendered like this:

{ "communalTable": { "capacity": 42 } }

Often in the real world you'll have to conform to a particular protocol format, or, even if that's not the case, being able to control the shape of the wire format is important to deal with backwards compatibility.

As I outlined in the introduction article you can usually find a more weakly typed API to get the job done. For serializing Table to JSON it looks like this:

public static string Serialize(this Table table)
{
    return table.Accept(new TableVisitor());
}
 
private sealed class TableVisitor : ITableVisitor<string>
{
    public string VisitCommunal(NaturalNumber capacity)
    {
        var j = new JsonObject
        {
            ["communalTable"] = new JsonObject
            {
                ["capacity"] = (int)capacity
            }
        };
        return j.ToJsonString();
    }
 
    public string VisitSingle(NaturalNumber capacity, NaturalNumber value)
    {
        var j = new JsonObject
        {
            ["singleTable"] = new JsonObject
            {
                ["capacity"] = (int)capacity,
                ["minimalReservation"] = (int)value
            }
        };
        return j.ToJsonString();
    }
}

In order to separate concerns, I've defined this functionality in a new static class that references the Domain Model. The Serialize extension method uses a private Visitor to write two different JsonObject objects, using the JSON API's underlying Document Object Model (DOM).

JSON deserialization by hand #

You can also go the other way, and when it looks more complicated, it's because it is. When serializing an encapsulated value, not a lot can go wrong because the value is already valid. When deserializing a JSON string, on the other hand, all sorts of things can go wrong: It might not even be a valid string, or the string may not be valid JSON, or the JSON may not be a valid Table representation, or the values may be illegal, etc.

Since there are several values that explicitly must be integers, it makes sense to define a helper method to try to parse an integer:

private static int? TryInt(this JsonNode? node)
{
    if (node is null)
        return null;
    
    if (node.GetValueKind() != JsonValueKind.Number)
        return null;
 
    try
    {
        return (int)node;
    }
    catch (FormatException)
    {
        return null;
    }
}

I'm surprised that there's no built-in way to do that, but if there is, I couldn't find it.

With a helper method like that you can now implement the Deserialize method:

public static Table? Deserialize(string json)
{
    try
    {
        var node = JsonNode.Parse(json);
 
        var cnode = node?["communalTable"];
        if (cnode is { })
        {
            var capacity = cnode["capacity"].TryInt();
            if (capacity is null)
                return null;
            return Table.TryCreateCommunal(capacity.Value);
        }
 
        var snode = node?["singleTable"];
        if (snode is { })
        {
            var capacity = snode["capacity"].TryInt();
            var minimalReservation = snode["minimalReservation"].TryInt();
            if (capacity is null || minimalReservation is null)
                return null;
            return Table.TryCreateSingle(
                capacity.Value,
                minimalReservation.Value);
        }
 
        return null;
    }
    catch (JsonException)
    {
        return null;
    }
}

Since both serialisation and deserialization is based on string values, you should write automated tests that verify that the code works, and in fact, I did. Here are a few examples:

[Fact]
public void DeserializeSingleTableFor4()
{
    var json = """{"singleTable":{"capacity":4,"minimalReservation":3}}""";
    var actual = TableJson.Deserialize(json);
    Assert.Equal(Table.TryCreateSingle(4, 3), actual);
}
 
[Fact]
public void DeserializeNonTable()
{
    var json = """{"foo":42}""";
    var actual = TableJson.Deserialize(json);
    Assert.Null(actual);
}

Apart from using directives and namespace declaration this hand-written JSON capability requires 87 lines of code, although, to be fair, TryInt is a general-purpose method that ought to be part of the System.Text.Json API. Can we do better with static types and Reflection?

JSON serialisation based on types #

The static JsonSerializer class comes with Serialize<T> and Deserialize<T> methods that use Reflection to convert a statically typed object to and from JSON. You can define a type (a Data Transfer Object (DTO) if you will) and let Reflection do the hard work.

In Code That Fits in Your Head I explain how you're usually better off separating the role of serialization from the role of Domain Model. One way to do that is exactly by defining a DTO for serialisation, and let the Domain Model remain exclusively to model the rules of the application. The above Table type plays the latter role, so we need new DTO types:

public sealed class TableDto
{
    public CommunalTableDto? CommunalTable { get; set; }
    public SingleTableDto? SingleTable { get; set; }
}

public sealed class CommunalTableDto
{
    public int Capacity { get; set; }
}

public sealed class SingleTableDto
{
    public int Capacity { get; set; }
    public int MinimalReservation { get; set; }
}

One way to model a sum type with a DTO is to declare both cases as nullable fields. While it does allow illegal states to be representable (i.e. both kinds of tables defined at the same time, or none of them present) this is only par for the course at the application boundary.

While you can serialize values of that type, by default the generated JSON doesn't have the right format. Instead, a serialized communal table looks like this:

{
  "CommunalTable": { "Capacity": 42 },
  "SingleTable": null
}

There are two problems with the generated JSON document:

• The casing is wrong
• The null value shouldn't be there

None of those are too hard to address, but it does make the API a bit more awkward to use, as this test demonstrates:

[Fact]
public void SerializeCommunalTableViaReflection()
{
    var dto = new TableDto
    {
        CommunalTable = new CommunalTableDto { Capacity = 42 }
    };
 
    var actual = JsonSerializer.Serialize(
        dto,
        new JsonSerializerOptions
        {
            PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
            DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
        });
 
    Assert.Equal("""{"communalTable":{"capacity":42}}""", actual);
}

You can, of course, define this particular serialization behaviour as a reusable method, so it's not a problem that you can't address. I just wanted to include this, since it's part of the overall work that you have to do in order to make this work.

JSON deserialisation based on types #

To allow parsing of JSON into the above DTO the Reflection-based Deserialize method pretty much works out of the box, although again, it needs to be configured. Here's a passing test that demonstrates how that works:

[Fact]
public void DeserializeSingleTableViaReflection()
{
    var json = """{"singleTable":{"capacity":4,"minimalReservation":2}}""";
 
    var actual = JsonSerializer.Deserialize<TableDto>(
        json,
        new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase });
 
    Assert.Null(actual?.CommunalTable);
    Assert.Equal(4, actual?.SingleTable?.Capacity);
    Assert.Equal(2, actual?.SingleTable?.MinimalReservation);
}

There's only difference in casing, so you'd expect the Deserialize method to be a Tolerant Reader, but no. It's very particular about that, so the JsonNamingPolicy.CamelCase configuration is necessary. Perhaps the API designers found that explicit is better than implicit.

In any case, you could package that in a reusable Deserialize function that has all the options that are appropriate in a particular code context, so not a big deal. That takes care of actually writing and parsing JSON, but that's only half the battle. This only gives you a way to parse and serialize the DTO. What you ultimately want is to persist or dehydrate Table data.

Converting DTO to Domain Model, and vice versa #

As usual, converting a nice, encapsulated value to a more relaxed format is safe and trivial:

public static TableDto ToDto(this Table table)
{
    return table.Accept(new TableDtoVisitor());
}
 
private sealed class TableDtoVisitor : ITableVisitor<TableDto>
{
    public TableDto VisitCommunal(NaturalNumber capacity)
    {
        return new TableDto
        {
            CommunalTable = new CommunalTableDto
            {
                Capacity = (int)capacity
            }
        };
    }
 
    public TableDto VisitSingle(
        NaturalNumber capacity,
        NaturalNumber value)
    {
        return new TableDto
        {
            SingleTable = new SingleTableDto
            {
                Capacity = (int)capacity,
                MinimalReservation = (int)value
            }
        };
    }
}

Going the other way is fundamentally a parsing exercise:

public Table? TryParse()
{
    if (CommunalTable is { })
        return Table.TryCreateCommunal(CommunalTable.Capacity);
    if (SingleTable is { })
        return Table.TryCreateSingle(
            SingleTable.Capacity,
            SingleTable.MinimalReservation);
 
    return null;
}

Here, like in Code That Fits in Your Head, I've made that conversion an instance method on TableDto.

Such an operation may fail, so the result is a nullable Table object.

Let's take stock of the type-based alternative. It requires 58 lines of code, distributed over three DTO types and the two conversions ToDto and TryParse, but here I haven't counted configuration of Serialize and Deserialize, since I left that to each test case that I wrote. Since all of this code generally stays within 80 characters in line width, that would realistically add another 10 lines of code, for a total around 68 lines.

This is smaller than the DOM-based code, but not by much.

Conclusion #

In this article I've explored two alternatives for converting a well-encapsulated Domain Model to and from JSON. One option is to directly manipulate the DOM. Another option is take a more declarative approach and define types that model the shape of the JSON data, and then leverage type-based automation (here, Reflection) to automatically parse and write the JSON.

I've deliberately chosen a Domain Model with some constraints, in order to demonstrate how persisting a non-trivial data model might work. With that setup, writing 'loosely coupled' code directly against the DOM requires 87 lines of code, while taking advantage of type-based automation requires 68 lines of code. Again, Reflection seems 'easier' if you count lines of code, but the difference is marginal.

Comments

Using System.Text.Json, with and without Reflection.

This article is part of a short series of articles about serialization with and without Reflection. In this instalment I'll explore some options for serializing JSON with F# using the API built into .NET: System.Text.Json. I'm not going use Json.NET in this article, but I've done similar things with that library in the past, so what's here is, at least, somewhat generalizable.

Natural numbers #

Before we start investigating how to serialize to and from JSON, we must have something to serialize. As described in the introductory article we'd like to parse and write restaurant table configurations like this:

{
  "singleTable": {
    "capacity": 16,
    "minimalReservation": 10
  }
}

On the other hand, I'd like to represent the Domain Model in a way that encapsulates the rules governing the model, making illegal states unrepresentable.

As the first step, we observe that the numbers involved are all natural numbers. In F# it's both idiomatic and easy to define a predicative data type:

type NaturalNumber = private NaturalNumber of int

Since it's defined with a private constructor we need to also supply a way to create valid values of the type:

module NaturalNumber =
    let tryCreate n = if n < 1 then None else Some (NaturalNumber n)

In this, as well as the other articles in this series, I've chosen to model the potential for errors with Option values. I could also have chosen to use Result if I wanted to communicate information along the 'error channel', but sticking with Option makes the code a bit simpler. Not so much in F# or Haskell, but once we reach C#, applicative validation becomes complicated.

There's no loss of generality in this decision, since both Option and Result are applicative functors.

> NaturalNumber.tryCreate -1;;
val it: NaturalNumber option = None

> let x = NaturalNumber.tryCreate 42;;
val x: NaturalNumber option = Some NaturalNumber 42

The tryCreate function enables client developers to create NaturalNumber values, and due to the F#'s default equality and comparison implementation, you can even compare them:

> let y = NaturalNumber.tryCreate 2112;;
val y: NaturalNumber option = Some NaturalNumber 2112

> x < y;;
val it: bool = true

That's it for natural numbers. Three lines of code. Compare that to the Haskell implementation, which required eight lines of code. This is mostly due to F#'s private keyword, which Haskell doesn't have.

Domain Model #

Modelling a restaurant table follows in the same vein. One invariant I would like to enforce is that for a 'single' table, the minimal reservation should be a NaturalNumber less than or equal to the table's capacity. It doesn't make sense to configure a table for four with a minimum reservation of six.

In the same spirit as above, then, define this type:

type Table =
    private
    | SingleTable of NaturalNumber * NaturalNumber
    | CommunalTable of NaturalNumber

Once more the private keyword makes it impossible for client code to create instances directly, so we need a pair of functions to create values:

module Table =
    let trySingle capacity minimalReservation = option {
        let! cap = NaturalNumber.tryCreate capacity
        let! min = NaturalNumber.tryCreate minimalReservation
        if cap < min then return! None
        else return SingleTable (cap, min) }
 
    let tryCommunal = NaturalNumber.tryCreate >> Option.map CommunalTable

Notice that trySingle checks the invariant that the capacity must be greater than or equal to the minimalReservation.

Again, notice how much easier it is to define a predicative type in F#, compared to Haskell.

This isn't a competition between languages, and while F# certainly scores a couple of points here, Haskell has other advantages.

The point of this little exercise, so far, is that it encapsulates the contract implied by the Domain Model. It does this by using the static type system to its advantage.

JSON serialization by hand #

At the boundaries of applications, however, there are no static types. Is the static type system still useful in that situation?

For a long time, the most popular .NET library for JSON serialization was Json.NET, but these days I find the built-in API offered in the System.Text.Json namespace adequate. This is also the case here.

The original rationale for this article series was to demonstrate how serialization can be done without Reflection, so I'll start there and return to Reflection later.

In this article series, I consider the JSON format fixed. A single table should be rendered as shown above, and a communal table should be rendered like this:

{ "communalTable": { "capacity": 42 } }

Often in the real world you'll have to conform to a particular protocol format, or, even if that's not the case, being able to control the shape of the wire format is important to deal with backwards compatibility.

As I outlined in the introduction article you can usually find a more weakly typed API to get the job done. For serializing Table to JSON it looks like this:

let serializeTable = function
    | SingleTable (NaturalNumber capacity, NaturalNumber minimalReservation) ->
        let j = JsonObject ()
        j["singleTable"] <- JsonObject ()
        j["singleTable"]["capacity"] <- capacity
        j["singleTable"]["minimalReservation"] <- minimalReservation
        j.ToJsonString ()
    | CommunalTable (NaturalNumber capacity) ->
        let j = JsonObject ()
        j["communalTable"] <- JsonObject ()
        j["communalTable"]["capacity"] <- capacity
        j.ToJsonString ()

In order to separate concerns, I've defined this functionality in a new module that references the module that defines the Domain Model. The serializeTable function pattern-matches on SingleTable and CommunalTable to write two different JsonObject objects, using the JSON API's underlying Document Object Model (DOM).

JSON deserialization by hand #

You can also go the other way, and when it looks more complicated, it's because it is. When serializing an encapsulated value, not a lot can go wrong because the value is already valid. When deserializing a JSON string, on the other hand, all sorts of things can go wrong: It might not even be a valid string, or the string may not be valid JSON, or the JSON may not be a valid Table representation, or the values may be illegal, etc.

Here I found it appropriate to first define a small API of parsing functions, mostly in order to make the object-oriented API more composable. First, I need some code that looks at the root JSON object to determine which kind of table it is (if it's a table at all). I found it appropriate to do that as a pair of active patterns:

let private (|Single|_|) (node : JsonNode) =
    match node["singleTable"] with
    | null -> None
    | tn -> Some tn
 
let private (|Communal|_|) (node : JsonNode) =
    match node["communalTable"] with
    | null -> None
    | tn -> Some tn

It turned out that I also needed a function to even check if a string is a valid JSON document:

let private tryParseJson (candidate : string) =
    try JsonNode.Parse candidate |> Some
    with | :? System.Text.Json.JsonException -> None

If there's a way to do that without a try/with expression, I couldn't find it. Likewise, trying to parse an integer turns out to be surprisingly complicated:

let private tryParseInt (node : JsonNode) =
    match node with
    | null -> None
    | _ ->
        if node.GetValueKind () = JsonValueKind.Number
        then
            try node |> int |> Some
            with | :? FormatException -> None // Thrown on decimal numbers
        else None

Both tryParseJson and tryParseInt are, however, general-purpose functions, so if you have a lot of JSON you need to parse, you can put them in a reusable library.

With those building blocks you can now define a function to parse a Table:

let tryDeserializeTable (candidate : string) =
    match tryParseJson candidate with
    | Some (Single node) -> option {
        let! capacity = node["capacity"] |> tryParseInt
        let! minimalReservation = node["minimalReservation"] |> tryParseInt
        return! Table.trySingle capacity minimalReservation }
    | Some (Communal node) -> option {
        let! capacity = node["capacity"] |> tryParseInt
        return! Table.tryCommunal capacity }
    | _ -> None

Since both serialisation and deserialization is based on string values, you should write automated tests that verify that the code works, and in fact, I did. Here are a few examples:

[<Fact>]
let ``Deserialize single table for 4`` () =
    let json = """{"singleTable":{"capacity":4,"minimalReservation":3}}"""
    let actual = tryDeserializeTable json
    Table.trySingle 4 3 =! actual
 
[<Fact>]
let ``Deserialize non-table`` () =
    let json = """{"foo":42}"""
    let actual = tryDeserializeTable json
    None =! actual

Apart from module declaration and imports etc. this hand-written JSON capability requires 46 lines of code, although, to be fair, some of that code (tryParseJson and tryParseInt) are general-purpose functions that belong in a reusable library. Can we do better with static types and Reflection?

JSON serialisation based on types #

The static JsonSerializer class comes with Serialize<T> and Deserialize<T> methods that use Reflection to convert a statically typed object to and from JSON. You can define a type (a Data Transfer Object (DTO) if you will) and let Reflection do the hard work.

In Code That Fits in Your Head I explain how you're usually better off separating the role of serialization from the role of Domain Model. One way to do that is exactly by defining a DTO for serialisation, and let the Domain Model remain exclusively to model the rules of the application. The above Table type plays the latter role, so we need new DTO types:

type CommunalTableDto = { Capacity : int }
type SingleTableDto = { Capacity : int; MinimalReservation : int }
type TableDto = {
    CommunalTable : CommunalTableDto option
    SingleTable : SingleTableDto option }

One way to model a sum type with a DTO is to declare both cases as option fields. While it does allow illegal states to be representable (i.e. both kinds of tables defined at the same time, or none of them present) this is only par for the course at the application boundary.

While you can serialize values of that type, by default the generated JSON doesn't have the right format:

> val dto: TableDto = { CommunalTable = Some { Capacity = 42 }
                        SingleTable = None }

> JsonSerializer.Serialize dto;;
val it: string = "{"CommunalTable":{"Capacity":42},"SingleTable":null}"

There are two problems with the generated JSON document:

• The casing is wrong
• The null value shouldn't be there

None of those are too hard to address, but it does make the API a bit more awkward to use, as this test demonstrates:

[<Fact>]
let ``Serialize communal table via reflection`` () =
    let dto = { CommunalTable = Some { Capacity = 42 }; SingleTable = None }
    let actual =
        JsonSerializer.Serialize (
            dto,
            JsonSerializerOptions (
                PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
                IgnoreNullValues = true ))
    """{"communalTable":{"capacity":42}}""" =! actual

You can, of course, define this particular serialization behaviour as a reusable function, so it's not a problem that you can't address. I just wanted to include this, since it's part of the overall work that you have to do in order to make this work.

JSON deserialisation based on types #

To allow parsing of JSON into the above DTO the Reflection-based Deserialize method pretty much works out of the box, although again, it needs to be configured. Here's a passing test that demonstrates how that works:

[<Fact>]
let ``Deserialize single table via reflection`` () =
    let json = """{"singleTable":{"capacity":4,"minimalReservation":2}}"""
    let actual =
        JsonSerializer.Deserialize<TableDto> (
            json,
            JsonSerializerOptions ( PropertyNamingPolicy = JsonNamingPolicy.CamelCase ))
    {
        CommunalTable = None
        SingleTable = Some { Capacity = 4; MinimalReservation = 2 }
    } =! actual

There's only difference in casing, so you'd expect the Deserialize method to be a Tolerant Reader, but no. It's very particular about that, so the JsonNamingPolicy.CamelCase configuration is necessary. Perhaps the API designers found that explicit is better than implicit.

In any case, you could package that in a reusable Deserialize function that has all the options that are appropriate in a particular code context, so not a big deal. That takes care of actually writing and parsing JSON, but that's only half the battle. This only gives you a way to parse and serialize the DTO. What you ultimately want is to persist or dehydrate Table data.

Converting DTO to Domain Model, and vice versa #

As usual, converting a nice, encapsulated value to a more relaxed format is safe and trivial:

let toTableDto = function
    | SingleTable (NaturalNumber capacity, NaturalNumber minimalReservation) ->
        {
            CommunalTable = None
            SingleTable =
                Some
                    {
                        Capacity = capacity
                        MinimalReservation = minimalReservation
                    }
        }
    | CommunalTable (NaturalNumber capacity) ->
        { CommunalTable = Some { Capacity = capacity }; SingleTable = None }

Going the other way is fundamentally a parsing exercise:

let tryParseTableDto candidate =
    match candidate.CommunalTable, candidate.SingleTable with
    | Some { Capacity = capacity }, None -> Table.tryCommunal capacity
    | None, Some { Capacity = capacity; MinimalReservation = minimalReservation } ->
        Table.trySingle capacity minimalReservation
    | _ -> None

Such an operation may fail, so the result is a Table option. It could also have been a Result<Table, 'something>, if you wanted to return information about errors when things go wrong. It makes the code marginally more complex, but doesn't change the overall thrust of this exploration.

Ironically, while tryParseTableDto is actually more complex than toTableDto it looks smaller, or at least denser.

Let's take stock of the type-based alternative. It requires 26 lines of code, distributed over three DTO types and the two conversions tryParseTableDto and toTableDto, but here I haven't counted configuration of Serialize and Deserialize, since I left that to each test case that I wrote. Since all of this code generally stays within 80 characters in line width, that would realistically add another 10 lines of code, for a total around 36 lines.

This is smaller than the DOM-based code, although at the same magnitude.

Conclusion #

In this article I've explored two alternatives for converting a well-encapsulated Domain Model to and from JSON. One option is to directly manipulate the DOM. Another option is take a more declarative approach and define types that model the shape of the JSON data, and then leverage type-based automation (here, Reflection) to automatically parse and write the JSON.

I've deliberately chosen a Domain Model with some constraints, in order to demonstrate how persisting a non-trivial data model might work. With that setup, writing 'loosely coupled' code directly against the DOM requires 46 lines of code, while taking advantage of type-based automation requires 36 lines of code. Contrary to the Haskell example, Reflection does seem to edge out a win this round.

Next: Serializing restaurant tables in C#.

Using Aeson, with and without generics.

This article is part of a short series of articles about serialization with and without Reflection. In this instalment I'll explore some options for serializing JSON using Aeson.

The source code is available on GitHub.

Natural numbers #

Before we start investigating how to serialize to and from JSON, we must have something to serialize. As described in the introductory article we'd like to parse and write restaurant table configurations like this:

{
  "singleTable": {
    "capacity": 16,
    "minimalReservation": 10
  }
}

On the other hand, I'd like to represent the Domain Model in a way that encapsulates the rules governing the model, making illegal states unrepresentable.

As the first step, we observe that the numbers involved are all natural numbers. While I'm aware that Haskell has built-in Nat type, I choose not to use it here, for a couple of reasons. One is that Nat is intended for type-level programming, and while this might be useful here, I don't want to pull in more exotic language features than are required. Another reason is that, in this domain, I want to model natural numbers as excluding zero (and I honestly don't remember if Nat allows zero, but I think that it does..?).

Another option is to use Peano numbers, but again, for didactic reasons, I'll stick with something a bit more idiomatic.

You can easily introduce a wrapper over, say, Integer, to model natural numbers:

newtype Natural = Natural Integer deriving (Eq, Ord, Show)

This, however, doesn't prevent you from writing Natural (-1), so we need to make this a predicative data type. The first step is to only export the type, but not its data constructor:

module Restaurants (
  Natural,
  -- More exports here...
  ) where

But this makes it impossible for client code to create values of the type, so we need to supply a smart constructor:

tryNatural :: Integer -> Maybe Natural
tryNatural n
  | n < 1 = Nothing
  | otherwise = Just (Natural n)

In this, as well as the other articles in this series, I've chosen to model the potential for errors with Maybe values. I could also have chosen to use Either if I wanted to communicate information along the 'error channel', but sticking with Maybe makes the code a bit simpler. Not so much in Haskell or F#, but once we reach C#, applicative validation becomes complicated.

There's no loss of generality in this decision, since both Maybe and Either are Applicative instances.

With the tryNatural function you can now (attempt to) create Natural values:

ghci> tryNatural (-1)
Nothing
ghci> x = tryNatural 42
ghci> x
Just (Natural 42)

This enables client developers to create Natural values, and due to the type's Ord instance, you can even compare them:

ghci> y = tryNatural 2112
ghci> x < y
True

Even so, there will be cases when you need to extract the underlying Integer from a Natural value. You could supply a normal function for that purpose, but in order to make some of the following code a little more elegant, I chose to do it with pattern synonyms:

{-# COMPLETE N #-}
pattern N :: Integer -> Natural
pattern N i <- Natural i

That needs to be exported as well.

So, eight lines of code to declare a predicative type that models a natural number. Incidentally, this'll be 2-3 lines of code in F#.

Domain Model #

Modelling a restaurant table follows in the same vein. One invariant I would like to enforce is that for a 'single' table, the minimal reservation should be a Natural number less than or equal to the table's capacity. It doesn't make sense to configure a table for four with a minimum reservation of six.

In the same spirit as above, then, define this type:

data SingleTable = SingleTable
  { singleCapacity :: Natural
  , minimalReservation :: Natural
  } deriving (Eq, Ord, Show)

Again, only export the type, but not its data constructor. In order to extract values, then, supply another pattern synonym:

{-# COMPLETE SingleT #-}
pattern SingleT :: Natural -> Natural -> SingleTable
pattern SingleT c m <- SingleTable c m

Finally, define a Table type and two smart constructors:

data Table = Single SingleTable | Communal Natural deriving (Eq, Show)
 
trySingleTable :: Integer -> Integer -> Maybe Table
trySingleTable capacity minimal = do
  c <- tryNatural capacity
  m <- tryNatural minimal
  if c < m then Nothing else Just (Single (SingleTable c m))
 
tryCommunalTable :: Integer -> Maybe Table
tryCommunalTable = fmap Communal . tryNatural

Notice that trySingleTable checks the invariant that the capacity must be greater than or equal to the minimal reservation.

The point of this little exercise, so far, is that it encapsulates the contract implied by the Domain Model. It does this by using the static type system to its advantage.

JSON serialization by hand #

At the boundaries of applications, however, there are no static types. Is the static type system still useful in that situation?

For Haskell, the most common JSON library is Aeson, and I admit that I'm no expert. Thus, it's possible that there's an easier way to serialize to and deserialize from JSON. If so, please leave a comment explaining the alternative.

The original rationale for this article series was to demonstrate how serialization can be done without Reflection, or, in the case of Haskell, Generics (not to be confused with .NET generics, which in Haskell usually is called parametric polymorphism). We'll return to Generics later in this article.

In this article series, I consider the JSON format fixed. A single table should be rendered as shown above, and a communal table should be rendered like this:

{ "communalTable": { "capacity": 42 } }

Often in the real world you'll have to conform to a particular protocol format, or, even if that's not the case, being able to control the shape of the wire format is important to deal with backwards compatibility.

As I outlined in the introduction article you can usually find a more weakly typed API to get the job done. For serializing Table to JSON it looks like this:

newtype JSONTable = JSONTable Table deriving (Eq, Show)
 
instance ToJSON JSONTable where
  toJSON (JSONTable (Single (SingleT (N c) (N m)))) =
    object ["singleTable" .= object [
      "capacity" .= c,
      "minimalReservation" .= m]]
  toJSON (JSONTable (Communal (N c))) =
    object ["communalTable" .= object ["capacity" .= c]]

In order to separate concerns, I've defined this functionality in a new module that references the module that defines the Domain Model. Thus, to avoid orphan instances, I've defined a JSONTable newtype wrapper that I then make a ToJSON instance.

The toJSON function pattern-matches on Single and Communal to write two different Values, using Aeson's underlying Document Object Model (DOM).

JSON deserialization by hand #

You can also go the other way, and when it looks more complicated, it's because it is. When serializing an encapsulated value, not a lot can go wrong because the value is already valid. When deserializing a JSON string, on the other hand, all sorts of things can go wrong: It might not even be a valid string, or the string may not be valid JSON, or the JSON may not be a valid Table representation, or the values may be illegal, etc.

It's no surprise, then, that the FromJSON instance is bigger:

instance FromJSON JSONTable where
  parseJSON (Object v) = do
    single <- v .:? "singleTable"
    communal <- v .:? "communalTable"
    case (single, communal) of
      (Just s, Nothing) -> do
        capacity <- s .: "capacity"
        minimal <- s .: "minimalReservation"
        case trySingleTable capacity minimal of
          Nothing -> fail "Expected natural numbers."
          Just t -> return $ JSONTable t
      (Nothing, Just c) -> do
        capacity <- c .: "capacity"
        case tryCommunalTable capacity of
          Nothing -> fail "Expected a natural number."
          Just t -> return $ JSONTable t
      _ -> fail "Expected exactly one of singleTable or communalTable."
  parseJSON _ = fail "Expected an object."

I could probably have done this more succinctly if I'd spent even more time on it than I already did, but it gets the job done and demonstrates the point. Instead of relying on run-time Reflection, the FromJSON instance is, unsurprisingly, a parser, composed from Aeson's specialised parser combinator API.

Since both serialisation and deserialization is based on string values, you should write automated tests that verify that the code works.

Apart from module declaration and imports etc. this hand-written JSON capability requires 27 lines of code. Can we do better with static types and Generics?

JSON serialisation based on types #

The intent with the Aeson library is that you define a type (a Data Transfer Object (DTO) if you will), and then let 'compiler magic' do the rest. In Haskell, it's not run-time Reflection, but a compilation technology called Generics. As I understand it, it automatically 'writes' the serialization and parsing code and turns it into machine code as part of normal compilation.

You're supposed to first turn on the

{-# LANGUAGE DeriveGeneric #-}

language pragma and then tell the compiler to automatically derive Generic for the DTO in question. You'll see an example of that shortly.

It's a fairly flexible system that you can tweak in various ways, but if it's possible to do it directly with the above Table type, please leave a comment explaining how. I tried, but couldn't make it work. To be clear, I could make it serializable, but not to the above JSON format. After enough Aeson Whac-A-Mole I decided to change tactics.

In Code That Fits in Your Head I explain how you're usually better off separating the role of serialization from the role of Domain Model. The way to do that is exactly by defining a DTO for serialisation, and let the Domain Model remain exclusively to model the rules of the application. The above Table type plays the latter role, so we need new DTO types.

We may start with the building blocks:

newtype CommunalDTO = CommunalDTO
  { communalCapacity :: Integer
  } deriving (Eq, Show, Generic)

Notice how it declaratively derives Generic, which works because of the DeriveGeneric language pragma.

From here, in principle, all that you need is just a single declaration to make it serializable:

instance ToJSON CommunalDTO

While it does serialize to JSON, it doesn't have the right format:

{ "communalCapacity": 42 }

The property name should be capacity, not communalCapacity. Why did I call the record field communalCapacity instead of capacity? Can't I just fix my CommunalDTO record?

Unfortunately, I can't just do that, because I also need a capacity JSON property for the single-table case, and Haskell isn't happy about duplicated field names in the same module. (This language feature truly is one of the weak points of Haskell.)

Instead, I can tweak the Aeson rules by supplying an Options value to the instance definition:

communalJSONOptions :: Options
communalJSONOptions =
  defaultOptions {
    fieldLabelModifier = \s -> case s of
      "communalCapacity" -> "capacity"
      _ -> s }
 
instance ToJSON CommunalDTO where
  toJSON = genericToJSON communalJSONOptions
  toEncoding = genericToEncoding communalJSONOptions

This instructs the compiler to modify how it generates the serialization code, and the generated JSON fragment is now correct.

We can do the same with the single-table case:

data SingleDTO = SingleDTO
  { singleCapacity :: Integer
  , minimalReservation :: Integer
  } deriving (Eq, Show, Generic)
 
singleJSONOptions :: Options
singleJSONOptions =
  defaultOptions {
    fieldLabelModifier = \s -> case s of
      "singleCapacity" -> "capacity"
      "minimalReservation" -> "minimalReservation"
      _ -> s }
 
instance ToJSON SingleDTO where
  toJSON = genericToJSON singleJSONOptions
  toEncoding = genericToEncoding singleJSONOptions

This takes care of that case, but we still need a container type that will hold either one or the other:

data TableDTO = TableDTO
  { singleTable :: Maybe SingleDTO
  , communalTable :: Maybe CommunalDTO
  } deriving (Eq, Show, Generic)
 
tableJSONOptions :: Options
tableJSONOptions =
  defaultOptions { omitNothingFields = True }
 
instance ToJSON TableDTO where
  toJSON = genericToJSON tableJSONOptions
  toEncoding = genericToEncoding tableJSONOptions

One way to model a sum type with a DTO is to declare both cases as Maybe fields. While it does allow illegal states to be representable (i.e. both kinds of tables defined at the same time, or none of them present) this is only par for the course at the application boundary.

That's quite a bit of infrastructure to stand up, but at least most of it can be reused for parsing.

JSON deserialisation based on types #

To allow parsing of JSON into the above DTO we can make them all FromJSON instances, e.g.:

instance FromJSON CommunalDTO where
  parseJSON = genericParseJSON communalJSONOptions

Notice that you can reuse the same communalJSONOptions used for the ToJSON instance. Repeat that exercise for the two other record types.

That's only half the battle, though, since this only gives you a way to parse and serialize the DTO. What you ultimately want is to persist or dehydrate Table data.

Converting DTO to Domain Model, and vice versa #

As usual, converting a nice, encapsulated value to a more relaxed format is safe and trivial:

toTableDTO :: Table -> TableDTO
toTableDTO (Single (SingleT (N c) (N m))) = TableDTO (Just (SingleDTO c m)) Nothing
toTableDTO (Communal (N c)) = TableDTO Nothing (Just (CommunalDTO c))

Going the other way is fundamentally a parsing exercise:

tryParseTable :: TableDTO -> Maybe Table
tryParseTable (TableDTO (Just (SingleDTO c m)) Nothing) = trySingleTable c m
tryParseTable (TableDTO Nothing (Just (CommunalDTO c))) = tryCommunalTable c
tryParseTable _ = Nothing

Such an operation may fail, so the result is a Maybe Table. It could also have been an Either something Table, if you wanted to return information about errors when things go wrong. It makes the code marginally more complex, but doesn't change the overall thrust of this exploration.

Let's take stock of the type-based alternative. It requires 62 lines of code, distributed over three DTO types, their Options, their ToJSON and FromJSON instances, and finally the two conversions tryParseTable and toTableDTO.

Conclusion #

In this article I've explored two alternatives for converting a well-encapsulated Domain Model to and from JSON. One option is to directly manipulate the DOM. Another option is take a more declarative approach and define types that model the shape of the JSON data, and then leverage type-based automation (here, Generics) to automatically produce the code that parses and writes the JSON.

I've deliberately chosen a Domain Model with some constraints, in order to demonstrate how persisting a non-trivial data model might work. With that setup, writing 'loosely coupled' code directly against the DOM requires 27 lines of code, while 'taking advantage' of type-based automation requires 62 lines of code.

To be fair, the dice don't always land that way. You can't infer a general rule from a single example, and it's possible that I could have done something clever with Aeson to reduce the code. Even so, I think that there's a conclusion to be drawn, and it's this:

Type-based automation (Generics, or run-time Reflection) may seem simple at first glance. Just declare a type and let some automation library do the rest. It may happen, however, that you need to tweak the defaults so much that it would be easier skipping the type-based approach and instead directly manipulating the DOM.

I love static type systems, but I'm also watchful of their limitations. There's likely to be an inflection point where, on the one side, a type-based declarative API is best, while on the other side of that point, a more 'lightweight' approach is better.

The position of such an inflection point will vary from context to context. Just be aware of the possibility, and explore alternatives if things begin to feel awkward.

Next: Serializing restaurant tables in F#.

An investigation of alternatives.

I recently wrote a tweet that caused more responses than usual:

"A decade ago, I used .NET Reflection so often that I know most the the API by heart.

"Since then, I've learned better ways to solve my problems. I can't remember when was the last time I used .NET Reflection. I never need it.

"Do you?"

Most people who read my tweets are programmers, and some are, perhaps, not entirely neurotypical, but I intended the last paragraph to be a rhetorical question. My point, really, was to point out that if I tell you it's possible to do without Reflection, one or two readers might keep that in mind and at least explore options the next time the urge to use Reflection arises.

A common response was that Reflection is useful for (de)serialization of data. These days, the most common case is going to and from JSON, but the problem is similar if the format is XML, CSV, or another format. In a sense, even reading to and from a database is a kind of serialization.

In this little series of articles, I'm going to explore some alternatives to Reflection. I'll use the same example throughout, and I'll stick to JSON, but you can easily extrapolate to other serialization formats.

Table layouts #

As always, I find the example domain of online restaurant reservation systems to be so rich as to furnish a useful example. Imagine a multi-tenant service that enables restaurants to take and manage reservations.

When a new reservation request arrives, the system has to make a decision on whether to accept or reject the request. The layout, or configuration, of tables plays a role in that decision.

Such a multi-tenant system may have an API for configuring the restaurant; essentially, entering data into the system about the size and policies regarding tables in a particular restaurant.

Most restaurants have 'normal' tables where, if you reserve a table for three, you'll have the entire table for a duration. Some restaurants also have one or more communal tables, typically bar seating where you may get a view of the kitchen. Quite a few high-end restaurants have tables like these, because it enables them to cater to single diners without reserving an entire table that could instead have served two paying customers.

In Copenhagen, on the other hand, it's also not uncommon to have a special room for larger parties. I think this has something to do with the general age of the buildings in the city. Most establishments are situated in older buildings, with all the trappings, including load-bearing walls, cellars, etc. As part of a restaurant's location, there may be a big cellar room, second-story room, or other room that's not practical for the daily operation of the place, but which works for parties of, say, 15-30 people. Such 'private dining' rooms can be used for private occasions or company outings.

A maître d'hôtel may wish to configure the system with a variety of tables, including communal tables, and private dining tables as described above.

One way to model such requirements is to distinguish between two kinds of tables: Communal tables, and 'single' tables, and where single tables come with an additional property that models the minimal reservation required to reserve that table. A JSON representation might look like this:

{
  "singleTable": {
    "capacity": 16,
    "minimalReservation": 10
  }
}

This may represent a private dining table that seats up to sixteen people, and where the maître d'hôtel has decided to only accept reservations for at least ten guests.

A singleTable can also be used to model 'normal' tables without special limits. If the restaurant has a table for four, but is ready to accept a reservation for one person, you can configure a table for four, with a minimum reservation of one.

Communal tables are different, though:

{ "communalTable": { "capacity": 10 } }

Why not just model that as ten single tables that each seat one?

You don't want to do that because you want to make sure that parties can eat together. Some restaurants have more than one communal table. Imagine that you only have two communal tables of ten seats each. What happens if you model this as twenty single-person tables?

If you do that, you may accept reservations for parties of six, six, and six, because 6 + 6 + 6 = 18 < 20. When those three groups arrive, however, you discover that you have to split one of the parties! The party getting separated may not like that at all, and you are, after all, in the hospitality business.

Exploration #

In each article in this short series, I'll explore serialization with and without Reflection in a few languages. I'll start with Haskell, since that language doesn't have run-time Reflection. It does have a related facility called generics, not to be confused with .NET or Java generics, which in Haskell are called parametric polymorphism. It's confusing, I know.

Haskell generics look a bit like .NET Reflection, and there's some overlap, but it's not quite the same. The main difference is that Haskell generic programming all 'resolves' at compile time, so there's no run-time Reflection in Haskell.

If you don't care about Haskell, you can skip that article.

• Serializing restaurant tables in Haskell
• Serializing restaurant tables in F#
• Serializing restaurant tables in C#

As you can see, the next article repeats the exercise in F#, and if you also don't care about that language, you can skip that article as well.

The C# article, on the other hand, should be readable to not only C# programmers, but also developers who work in sufficiently equivalent languages.

Descriptive, not prescriptive #

The purpose of this article series is only to showcase alternatives. Based on the reactions my tweet elicited I take it that some people can't imagine how serialisation might look without Reflection.

It is not my intent that you should eschew the Reflection-based APIs available in your languages. In .NET, for example, a framework like ASP.NET MVC expects you to model JSON or XML as Data Transfer Objects. This gives you an illusion of static types at the boundary.

Even a Haskell web library like Servant expects you to model web APIs with static types.

When working with such a framework, it doesn't always pay to fight against its paradigm. When I work with ASP.NET, I define DTOs just like everyone else. On the other hand, if communicating with a backend system, I sometimes choose to skip static types and instead working directly with a JSON Document Object Model (DOM).

I occasionally find that it better fits my use case, but it's not the majority of times.

Conclusion #

While some sort of Reflection or metadata-driven mechanism is often used to implement serialisation, it often turns out that such convenient language capabilities are programmed on top of an ordinary object model. Even isolated to .NET, I think I'm on my third JSON library, and most (all?) turned out to have an underlying DOM that you can manipulate.

In this article I've set the stage for exploring how serialisation can work, with or (mostly) without Reflection.

If you're interested in the philosophy of science and epistemology, you may have noticed a recurring discussion in academia: A wider society benefits not only from learning what works, but also from learning what doesn't work. It would be useful if researchers published their failures along with their successes, yet few do (for fairly obvious reasons).

Well, I depend neither on research grants nor salary, so I'm free to publish negative results, such as they are.

Not that I want to go so far as to categorize what I present in the present articles as useless, but they're probably best applied in special circumstances. On the other hand, I don't know your context, and perhaps you're doing something I can't even imagine, and what I present here is just what you need.

Next: Serializing restaurant tables in Haskell.

Comments

            data MyType = MyType
              { myString :: String,
                myInt :: Int
              }
              deriving stock (Eq, Show, Typeable)
            

Or, rather: Try not to.

A few months ago I visited a customer and as the day was winding down we got to talk more informally. One of the architects mentioned, in an almost off-hand manner, "we've embarked on a SAFe journey..."

"Yes..?" I responded, hoping that my inflection would sound enough like a question that he'd elaborate.

Unfortunately, I'm apparently sometimes too subtle when dealing with people face-to-face, so I never got to hear just how that 'SAFe journey' was going. Instead, the conversation shifted to the adjacent topic of how to coordinate independent teams.

I told them that, in my opinion, the best way to coordinate independent teams is to not coordinate them. I don't remember exactly how I proceeded from there, but I probably said something along the lines that I consider coordination meetings between teams to be an 'architecture smell'. That the need to talk to other teams was a symptom that teams were too tightly coupled.

I don't remember if I said exactly that, but it would have been in character.

The architect responded: "I don't like silos."

How do you respond to that?

Autonomous teams #

I couldn't very well respond that silos are great. First, it doesn't sound very convincing. Second, it'd be an argument suitable only in a kindergarten. Are not! -Are too! -Not! -Too! etc.

After feeling momentarily checked, for once I managed to think on my feet, so I replied, "I don't suggest that your teams should be isolated from each other. I do encourage people to talk to each other, but I don't think that teams should coordinate much. Rather, think of each team as an organism on the savannah. They interact, and what they do impact others, but in the end they're autonomous life forms. I believe an architect's job is like a ranger's. You can't control the plants or animals, but you can nurture the ecosystem, herding it in a beneficial direction."

That ranger metaphor is an old pet peeve of mine, originating from what I consider one of my most under-appreciated articles: Zookeepers must become Rangers. It's closely related to the more popular metaphor of software architecture as gardening, but I like the wildlife variation because it emphasizes an even more hands-off approach. It removes the illusion that you can control a fundamentally unpredictable process, but replaces it with the hopeful emphasis on stewardship.

How do ecosystems thrive? A software architect (or ranger) should nurture resilience in each subsystem, just like evolution has promoted plants' and animals' ability to survive a variety of unforeseen circumstances: Flood, draught, fire, predators, lack of prey, disease, etc.

You want teams to work independently. This doesn't mean that they work in isolation, but rather they they are free to act according to their abilities and understanding of the situation. An architect can help them understand the wider ecosystem and predict tomorrow's weather, so to speak, but the team should remain autonomous.

Concurrent work #

I'm assuming that an organisation has multiple teams because they're supposed to work concurrently. While team A is off doing one thing, team B is doing something else. You can attempt to herd them in the same general direction, but beware of tight coordination.

What's the problem with coordination? Isn't it a kind of collaboration? Don't we consider that beneficial?

I'm not arguing that teams should be antagonistic. Like all metaphors, we should be careful not to take the savannah metaphor too far. I'm not imagining that one team consists of lions, apex predators, killing and devouring other teams.

Rather, the reason I'm wary of coordination is because it seems synonymous with synchronisation.

In Code That Fits in Your Head I've already discussed how good practices for Continuous Integration are similar to earlier lessons about optimistic concurrency. It recently struck me that we can draw a similar parallel between concurrent team work and parallel computing.

For decades we've known that the less synchronization, the faster parallel code is. Synchronization is costly.

In team work, coordination is like thread synchronization. Instead of doing work, you stop in order to coordinate. This implies that one thread or team has to wait for the other to catch up.

Unless work is perfectly evenly divided, team A may finish before team B. In order to coordinate, team A must sit idle for a while, waiting for B to catch up. (In development organizations, idleness is rarely allowed, so in practice, team A embarks on some other work, with consequences that I've already outlined.)

If you have more than two teams, this phenomenon only becomes worse. You'll have more idle time. This reminds me of Amdahl's law, which briefly put expresses that there's a limit to how much of a speed improvement you can get from concurrent work. The limit is related to the percentage of the work that can not be parallelized. The greater the need to synchronize work, the lower the ceiling. Conversely, the more you can let concurrent processes run without coordination, the more you gain from parallelization.

It seems to me that there's a direct counterpart in team organization. The more teams need to coordinate, the less is gained from having multiple teams.

But really, Fred Brooks could you have told you so in 1975.

Versioning #

A small development team may organize work informally. Work may be divided along 'natural' lines, each developer taking on tasks best suited to his or her abilities. If working in a code base with shared ownership, one developer doesn't have to wait on the work done by another developer. Instead, a programmer may complete the required work individually, or working together with a colleague. Coordination happens, but is both informal and frequent.

As development organizations grow, teams are formed. Separate teams are supposed to work independently, but may in practice often depend on each other. Team A may need team B to make a change before they can proceed with their own work. The (felt) need to coordinate team activities arise.

In my experience, this happens for a number of reasons. One is that teams may be divided along wrong lines; this is a socio-technical problem. Another, more technical, reason is that zookeepers rarely think explicitly about versioning or avoiding breaking changes. Imagine that team A needs team B to develop a new capability. This new capability implies a breaking change, so the teams will now need to coordinate.

Instead, team B should develop the new feature in such a way that it doesn't break existing clients. If all else fails, the new feature must exist side-by-side with the old way of doing things. With Continuous Deployment the new feature becomes available when it's ready. Team A still has to wait for the feature to become available, but no synchronization is required.

Conclusion #

Yet another lesson about thread-safety and concurrent transactions seems to apply to people and processes. Parallel processes should be autonomous, with as little synchronization as possible. The more you coordinate development teams, the more you limit the speed of overall work. This seems to suggest that something akin to Amdahl's law also applies to development organizations.

Instead of coordinating teams, encourage them to exist as autonomous entities, but set things up so that not breaking compatibility is a major goal for each team.

A refactoring example.

When I introduce the Fake Object testing pattern to people, a common concern is the maintenance burden of it. The point of the pattern is that you write some 'working' code only for test purposes. At a glance, it seems as though it'd be more work than using a dynamic mock library like Moq or Mockito.

This article isn't really about that, but the benefit of a Fake Object is that it has a lower maintenance footprint because it gives you a single class to maintain when you change interfaces or base classes. Dynamic mock objects, on the contrary, leads to Shotgun surgery because every time you change an interface or base class, you have to revisit multiple tests.

In a recent article I presented a Fake Object that may have looked bigger than most people would find comfortable for test code. In this article I discuss how to trim it via a set of refactorings.

Original Fake read registry #

The article presented this FakeReadRegistry, repeated here for your convenience:

internal sealed class FakeReadRegistry : IReadRegistry
{
    private readonly IReadOnlyCollection<Room> rooms;
    private readonly IDictionary<DateOnly, IReadOnlyCollection<Room>> views;
 
    public FakeReadRegistry(params Room[] rooms)
    {
        this.rooms = rooms;
        views = new Dictionary<DateOnly, IReadOnlyCollection<Room>>();
    }
 
    public IReadOnlyCollection<Room> GetFreeRooms(DateOnly arrival, DateOnly departure)
    {
        return EnumerateDates(arrival, departure)
            .Select(GetView)
            .Aggregate(rooms.AsEnumerable(), Enumerable.Intersect)
            .ToList();
    }
 
    public void RoomBooked(Booking booking)
    {
        foreach (var d in EnumerateDates(booking.Arrival, booking.Departure))
        {
            var view = GetView(d);
            var newView = QueryService.Reserve(booking, view);
            views[d] = newView;
        }
    }
 
    private static IEnumerable<DateOnly> EnumerateDates(DateOnly arrival, DateOnly departure)
    {
        var d = arrival;
        while (d < departure)
        {
            yield return d;
            d = d.AddDays(1);
        }
    }
 
    private IReadOnlyCollection<Room> GetView(DateOnly date)
    {
        if (views.TryGetValue(date, out var view))
            return view;
        else
            return rooms;
    }
}

This is 47 lines of code, spread over five members (including the constructor). Three of the methods have a cyclomatic complexity (CC) of 2, which is the maximum for this class. The remaining two have a CC of 1.

While you can play some CC golf with those CC-2 methods, that tends to pull the code in a direction of being less idiomatic. For that reason, I chose to present the code as above. Perhaps more importantly, it doesn't save that many lines of code.

Had this been a piece of production code, no-one would bat an eye at size or complexity, but this is test code. To add spite to injury, those 47 lines of code implement this two-method interface:

public interface IReadRegistry
{
    IReadOnlyCollection<Room> GetFreeRooms(DateOnly arrival, DateOnly departure);
 
    void RoomBooked(Booking booking);
}

Can we improve the situation?

Root cause analysis #

Before you rush to 'improve' code, it pays to understand why it looks the way it looks.

Code is a wonderfully malleable medium, so you should regard nothing as set in stone. On the other hand, there's often a reason it looks like it does. It may be that the previous programmers were incompetent ogres for hire, but often there's a better explanation.

I've outlined my thinking process in the previous article, and I'm not going to repeat it all here. To summarise, though, I've applied the Dependency Inversion Principle.

"clients [...] own the abstract interfaces"

In other words, I let the needs of the clients guide the design of the IReadRegistry interface, and then the implementation (FakeReadRegistry) had to conform.

But that's not the whole truth.

I was doing a programming exercise - the CQRS booking kata - and I was following the instructions given in the description. They quite explicitly outline the two dependencies and their methods.

When trying a new exercise, it's a good idea to follow instructions closely, so that's what I did. Once you get a sense of a kata, though, there's no law saying that you have to stick to the original rules. After all, the purpose of an exercise is to train, and in programming, trying new things is training.

Test code that wants to be production code #

A major benefit of test-driven development (TDD) is that it provides feedback. It pays to be tuned in to that channel. The above FakeReadRegistry seems to be trying to tell us something.

Consider the GetFreeRooms method. I'll repeat the single-expression body here for your convenience:

return EnumerateDates(arrival, departure)
    .Select(GetView)
    .Aggregate(rooms.AsEnumerable(), Enumerable.Intersect)
    .ToList();

Why is that the implementation? Why does it need to first enumerate the dates in the requested interval? Why does it need to call GetView for each date?

Why don't I just do the following and be done with it?

internal sealed class FakeStorage : Collection<Booking>, IWriteRegistry, IReadRegistry
{
    private readonly IReadOnlyCollection<Room> rooms;
 
    public FakeStorage(params Room[] rooms)
    {
        this.rooms = rooms;
    }
 
    public IReadOnlyCollection<Room> GetFreeRooms(DateOnly arrival, DateOnly departure)
    {
        var booked = this.Where(b => b.Overlaps(arrival, departure)).ToList();
        return rooms
            .Where(r => !booked.Any(b => b.RoomName == r.Name))
            .ToList();
    }
 
    public void Save(Booking booking)
    {
        Add(booking);
    }
}

To be honest, that's what I did first.

While there are two interfaces, there's only one Fake Object implementing both. That's often an easy way to address the Interface Segregation Principle and still keeping the Fake Object simple.

This is much simpler than FakeReadRegistry, so why didn't I just keep that?

I didn't feel it was an honest attempt at CQRS. In CQRS you typically write the data changes to one system, and then you have another logical process that propagates the information about the data modification to the read subsystem. There's none of that here. Instead of being based on one or more 'materialised views', the query is just that: A query.

That was what I attempted to address with FakeReadRegistry, and I think it's a much more faithful CQRS implementation. It's also more complex, as CQRS tends to be.

In both cases, however, it seems that there's some production logic trapped in the test code. Shouldn't EnumerateDates be production code? And how about the general 'algorithm' of RoomBooked:

• Enumerate the relevant dates
• Get the 'materialised' view for each date
• Calculate the new view for that date
• Update the collection of views for that date

That seems like just enough code to warrant moving it to the production code.

A word of caution before we proceed. When deciding to pull some of that test code into the production code, I'm making a decision about architecture.

Until now, I'd been following the Dependency Inversion Principle closely. The interfaces exist because the client code needs them. Those interfaces could be implemented in various ways: You could use a relational database, a document database, files, blobs, etc.

Once I decide to pull the above algorithm into the production code, I'm choosing a particular persistent data structure. This now locks the data storage system into a design where there's a persistent view per date, and another database of bookings.

Now that I'd learned some more about the exercise, I felt confident making that decision.

Template Method #

The first move I made was to create a superclass so that I could employ the Template Method pattern:

public abstract class ReadRegistry : IReadRegistry
{
    public IReadOnlyCollection<Room> GetFreeRooms(DateOnly arrival, DateOnly departure)
    {
        return EnumerateDates(arrival, departure)
            .Select(GetView)
            .Aggregate(Rooms.AsEnumerable(), Enumerable.Intersect)
            .ToList();
    }
 
    public void RoomBooked(Booking booking)
    {
        foreach (var d in EnumerateDates(booking.Arrival, booking.Departure))
        {
            var view = GetView(d);
            var newView = QueryService.Reserve(booking, view);
            UpdateView(d, newView);
        }
    }
 
    protected abstract void UpdateView(DateOnly date, IReadOnlyCollection<Room> view);
 
    protected abstract IReadOnlyCollection<Room> Rooms { get; }
 
    protected abstract bool TryGetView(DateOnly date, out IReadOnlyCollection<Room> view);
 
    private static IEnumerable<DateOnly> EnumerateDates(DateOnly arrival, DateOnly departure)
    {
        var d = arrival;
        while (d < departure)
        {
            yield return d;
            d = d.AddDays(1);
        }
    }
 
    private IReadOnlyCollection<Room> GetView(DateOnly date)
    {
        if (TryGetView(date, out var view))
            return view;
        else
            return Rooms;
    }
}

This looks similar to FakeReadRegistry, so how is this an improvement?

The new ReadRegistry class is production code. It can, and should, be tested. (Due to the history of how we got here, it's already covered by tests, so I'm not going to repeat that effort here.)

True to the Template Method pattern, three abstract members await a child class' implementation. These are the UpdateView and TryGetView methods, as well as the Rooms read-only property (glorified getter method).

Imagine that in the production code, these are implemented based on file/document/blob storage - one per date. TryGetView would attempt to read the document from storage, UpdateView would create or modify the document, while Rooms returns a default set of rooms.

A Test Double, however, can still use an in-memory dictionary:

internal sealed class FakeReadRegistry : ReadRegistry
{
    private readonly IReadOnlyCollection<Room> rooms;
    private readonly IDictionary<DateOnly, IReadOnlyCollection<Room>> views;
 
    protected override IReadOnlyCollection<Room> Rooms => rooms;
 
    public FakeReadRegistry(params Room[] rooms)
    {
        this.rooms = rooms;
        views = new Dictionary<DateOnly, IReadOnlyCollection<Room>>();
    }
 
    protected override void UpdateView(DateOnly date, IReadOnlyCollection<Room> view)
    {
        views[date] = view;
    }
 
    protected override bool TryGetView(DateOnly date, out IReadOnlyCollection<Room> view)
    {
        return views.TryGetValue(date, out view);
    }
}

Each override is a one-liner with cyclomatic complexity 1.

First round of clean-up #

An abstract class is already a polymorphic object, so we no longer need the IReadRegistry interface. Delete that, and update all code accordingly. Particularly, the QueryService now depends on ReadRegistry rather than IReadRegistry:

private readonly ReadRegistry readRegistry;
 
public QueryService(ReadRegistry readRegistry)
{
    this.readRegistry = readRegistry;
}

Now move the Reserve function from QueryService to ReadRegistry. Once this is done, the QueryService looks like this:

public sealed class QueryService
{
    private readonly ReadRegistry readRegistry;
 
    public QueryService(ReadRegistry readRegistry)
    {
        this.readRegistry = readRegistry;
    }
 
    public IReadOnlyCollection<Room> GetFreeRooms(DateOnly arrival, DateOnly departure)
    {
        return readRegistry.GetFreeRooms(arrival, departure);
    }
}

That class is only passing method calls along, so clearly no longer serving any purpose. Delete it.

This is a not uncommon in CQRS. One might even argue that if CQRS is done right, there's almost no code on the query side, since all the data view update happens as events propagate.

From abstract class to Dependency Injection #

While the current state of the code is based on an abstract base class, the overall architecture of the system doesn't hinge on inheritance. From Abstract class isomorphism we know that it's possible to refactor an abstract class to Constructor Injection. Let's do that.

First add an IViewStorage interface that mirrors the three abstract methods defined by ReadRegistry:

public interface IViewStorage
{
    IReadOnlyCollection<Room> Rooms { get; }
 
    void UpdateView(DateOnly date, IReadOnlyCollection<Room> view);
 
    bool TryGetView(DateOnly date, out IReadOnlyCollection<Room> view);
}

Then implement it with a Fake Object:

public sealed class FakeViewStorage : IViewStorage
{
    private readonly IDictionary<DateOnly, IReadOnlyCollection<Room>> views;
 
    public IReadOnlyCollection<Room> Rooms { get; }
 
    public FakeViewStorage(params Room[] rooms)
    {
        Rooms = rooms;
        views = new Dictionary<DateOnly, IReadOnlyCollection<Room>>();
    }
 
    public void UpdateView(DateOnly date, IReadOnlyCollection<Room> view)
    {
        views[date] = view;
    }
 
    public bool TryGetView(DateOnly date, out IReadOnlyCollection<Room> view)
    {
        return views.TryGetValue(date, out view);
    }
}

Notice the similarity to FakeReadRegistry, which we'll get rid of shortly.

Now inject IViewStorage into ReadRegistry, and make ReadRegistry a regular (sealed) class:

public sealed class ReadRegistry
{
    private readonly IViewStorage viewStorage;
 
    public ReadRegistry(IViewStorage viewStorage)
    {
        this.viewStorage = viewStorage;
    }
 
    public IReadOnlyCollection<Room> GetFreeRooms(DateOnly arrival, DateOnly departure)
    {
        return EnumerateDates(arrival, departure)
            .Select(GetView)
            .Aggregate(viewStorage.Rooms.AsEnumerable(), Enumerable.Intersect)
            .ToList();
    }
 
    public void RoomBooked(Booking booking)
    {
        foreach (var d in EnumerateDates(booking.Arrival, booking.Departure))
        {
            var view = GetView(d);
            var newView = Reserve(booking, view);
            viewStorage.UpdateView(d, newView);
        }
    }
 
    public static IReadOnlyCollection<Room> Reserve(
        Booking booking,
        IReadOnlyCollection<Room> existingView)
    {
        return existingView
            .Where(r => r.Name != booking.RoomName)
            .ToList();
    }
 
    private static IEnumerable<DateOnly> EnumerateDates(DateOnly arrival, DateOnly departure)
    {
        var d = arrival;
        while (d < departure)
        {
            yield return d;
            d = d.AddDays(1);
        }
    }
 
    private IReadOnlyCollection<Room> GetView(DateOnly date)
    {
        if (viewStorage.TryGetView(date, out var view))
            return view;
        else
            return viewStorage.Rooms;
    }
}

You can now delete the FakeReadRegistry Test Double, since FakeViewStorage has now taken its place.

Finally, we may consider if we can make FakeViewStorage even slimmer. While I usually favour composition over inheritance, I've found that deriving Fake Objects from collection base classes is often an efficient way to get a lot of mileage out of a few lines of code. FakeReadRegistry, however, had to inherit from ReadRegistry, so it couldn't derive from any other class.

FakeViewStorage isn't constrained in that way, so it's free to inherit from Dictionary<DateOnly, IReadOnlyCollection<Room>>:

public sealed class FakeViewStorage : Dictionary<DateOnly, IReadOnlyCollection<Room>>, IViewStorage
{
    public IReadOnlyCollection<Room> Rooms { get; }
 
    public FakeViewStorage(params Room[] rooms)
    {
        Rooms = rooms;
    }
 
    public void UpdateView(DateOnly date, IReadOnlyCollection<Room> view)
    {
        this[date] = view;
    }
 
    public bool TryGetView(DateOnly date, out IReadOnlyCollection<Room> view)
    {
        return TryGetValue(date, out view);
    }
}

This last move isn't strictly necessary, but I found it worth at least mentioning.

I hope you'll agree that this is a Fake Object that looks maintainable.

Conclusion #

Test-driven development is a feedback mechanism. If something is difficult to test, it tells you something about your System Under Test (SUT). If your test code looks bloated, that tells you something too. Perhaps part of the test code really belongs in the production code.

In this article, we started with a Fake Object that looked like it contained too much production code. Via a series of refactorings I moved the relevant parts to the production code, leaving me with a more idiomatic and conforming implementation.