ploeh blog danish software design
Roman numerals via property-based TDD
An example of doing the Roman numerals kata with property-based test-driven development.
The Roman numerals kata is a simple programming exercise. You should implement conversion to and from Roman numerals. This always struck me as the ultimate case for example-driven development, but I must also admit that I've managed to get stuck on the exercise doing exactly that: throwing more and more examples at the problem. Prompted by previous successes with property-based testing, I wondered whether the problem would be more tractable if approached with property-based testing. This turned out to be the case.
Single values #
When modelling a problem with property-based testing, you should attempt to express it in terms of general rules, but even so, the fundamental rule of Roman numerals is that there are certain singular symbols that have particular numeric values. There are no overall principles guiding these relationships; they simply are. Therefore, you'll still need to express these singular values as particular values. This is best done with a simple parametrised test, here using xUnit.net 2.1:
[<Theory>] [<InlineData("I", 1)>] [<InlineData("V", 5)>] [<InlineData("X", 10)>] [<InlineData("L", 50)>] [<InlineData("C", 100)>] [<InlineData("D", 500)>] [<InlineData("M", 1000)>] let ``elemental symbols have correct values`` (symbol : string) expected = Some expected =! Numeral.tryParseRoman symbol
The =! operator is a custom operator defined by Unquote (3.1.1), an assertion library. You can read it as must equal; that is, you can read this particular assertion as some expected must equal tryParseRoman symbol.
As you can see, this simple test expresses the relationship between the singular Roman numeral values and their decimal counterparts. You might still consider this automated test as example-driven, but I more think about it as establishing the ground rules for how Roman numerals work. If you look at the Wikipedia article, for instance, it also starts explaining the system by listing the values of these seven symbols.
Round-tripping #
A common technique when applying property-based testing to parsing problems is to require that values can round-trip. This should also be the case here:
let romanRange = Gen.elements [1..3999] |> Arb.fromGen [<Property(QuietOnSuccess = true)>] let ``tryParse is the inverse of toRoman`` () = Prop.forAll romanRange <| fun i -> test <@ Some i = (i |> Numeral.toRoman |> Option.bind Numeral.tryParseRoman) @>
This property uses FsCheck (2.2.4). First, it expresses a range of relevant integers. For various reasons (that we'll return to) we're only going to attempt conversion of the integers between 1 and 3,999. The value romanRange has the type Arbitrary<int>, where Arbitrary<'a> is a type defined by FsCheck. You can think of it as a generator of random values. In this case, romanRange generates random integers between 1 and 3,999.
When used with Prop.forAll, the property states that for all values drawn from romanRange, the anonymous function should succeed. The i argument within that anonymous function is populated by romanRange, and the function is executed 100 times (by default).
The test function is another Unquote function. It evaluates and reports on the quoted boolean expression; if it evaluates to true, nothing happens, but it throws an exception if it evaluates to false.
The particular expression states that if you call toRoman with i, and then call tryParseRoman with the return value of that function call, the result should be equal to i. Both sides should be wrapped in a Some case, though, since both toRoman and tryParseRoman might also return None. For the values in romanRange, however, you'd expect that the round-trip always succeeds.
Additivity #
The fundamental idea about Roman numerals is that they are additive: I means 1, II means (1 + 1 =) 2, XXX means (10 + 10 + 10 =) 30, and MLXVI means (1000 + 50 + 10 + 5 + 1 =) 1066. You simply count and add. Yes, there are special rules for subtractive shorthand, but forget about those for a moment. If you have a Roman numeral with symbols in strictly descending order, you can simply add the symbol values together.
You can express this with FsCheck. It looks a little daunting, but actually isn't that bad. I'll show it first, and then walk you through it:
[<Property(QuietOnSuccess = true)>] let ``symbols in descending order are additive`` () = let stringRepeat char count = String (char, count) let genSymbols count symbol = [0..count] |> List.map (stringRepeat symbol) |> Gen.elements let thousands = genSymbols 3 'M' let fiveHundreds = genSymbols 1 'D' let hundreds = genSymbols 3 'C' let fifties = genSymbols 1 'L' let tens = genSymbols 3 'X' let fives = genSymbols 1 'V' let ones = genSymbols 3 'I' let symbols = [thousands; fiveHundreds; hundreds; fifties; tens; fives; ones] |> Gen.sequence |> Gen.map String.Concat |> Arb.fromGen Prop.forAll symbols <| fun s -> let actual = Numeral.tryParseRoman s let expected = s |> Seq.map (string >> Numeral.tryParseRoman) |> Seq.choose id |> Seq.sum |> Some expected =! actual
The first two lines are two utility functions. The function stringRepeat has the type char -> int -> string. It simply provides a curried form of the String constructor overload that enables you to repeat a particular char value. As an example, stringRepeat 'I' 0 is "", stringRepeat 'X' 2 is "XX", and so on.
The function genSymbols has the type int -> char -> Gen<string>. It returns a generator that produces a repeated string no longer than the specified length. Thus, genSymbols 3 'M' is a generator that draws random values from the set [""; "M"; "MM"; "MMM"], genSymbols 1 'D' is a generator that draws random values from the set [""; "D"], and so on. Notice that the empty string is one of the values that the generator may use. This is by design.
Using genSymbols, you can define generators for all the symbols: up to three thousands, up to one five hundreds, up to three hundreds, etc. thousands, fiveHundreds, hundreds, and so on, are all values of the type Gen<string>.
You can combine all these string generators to a single generator using Gen.sequence, which takes a seq<Gen<'a>> as input and returns a Gen<'a list>. In this case, the input is [thousands; fiveHundreds; hundreds; fifties; tens; fives; ones], which has the type Gen<string> list, so the output is a Gen<string list> value. Values generated could include ["MM"; ""; ""; ""; "X"; ""; ""], [""; "D"; "CC"; "L"; "X"; "V"; ""], and so on.
Instead of lists of strings, you need single string values. These are easy to create using the built-in method String.Concat. You have to do it within a Gen value, though, so it's Gen.map String.Concat. Finally, you can convert the resulting Gen<string> to an Arbitrary using Arb.fromGen. The final symbols value has the type Arbitrary<string>. It'll generate values such as "MMX" and "DCCLXV".
This is a bit of work to ensure that proper Roman numerals are generated, but the rest of the property is tractable. You can use FsCheck's Prop.forAll to express the property that when tryParseRoman is called with any of the generated numerals, the return value is equal to the expected value.
The expected value is the sum of the value of each of the symbols in the input string, s. The string type implements the interface char seq, so you can map each of the characters by invoking tryParseRoman. That gives you a int option values. You can use Seq.choose id to throw away any None values there may be, and then Seq.sum to calculate the sum of the integers. Finally, you can pipe the sum into the Some case constructor to turn expected into an int option, which matches the type of actual.
Now that you have expected and actual values, you can assert that these two values must be equal to each other. This property states that for all strictly descending numerals, the return value must be the sum of the constituent symbols.
Subtractiveness #
The principal rule for Roman numerals is that of additivity, but if you only apply the above rules, you'd allow numerals such as IIII for 4, LXXXX for 90, etc. While there's historical precedent for such notation, it's not allowed in 'modern' Roman numerals. If there are more than three repeated characters, you should instead prefer subtractive notation: IV for 4, XC for 90, and so on.
Subtractive notation is, however, only allowed within adjacent groups. Logically, you could write 1999 as MIM, but this isn't allowed. The symbol I can only be used to subtract from V and X, X can only subtract from L and C, and C can only subtract from D and M.
Within these constraints, you have to describe the property of subtractive notation. Here's one way to do it:
type OptionBuilder() = member this.Bind(v, f) = Option.bind f v member this.Return v = Some v let opt = OptionBuilder() [<Property(QuietOnSuccess = true)>] let ``certain symbols in ascending are subtractive`` () = let subtractive (subtrahend : char) (minuends : string) = gen { let! minuend = Gen.elements minuends return subtrahend, minuend } let symbols = Gen.oneof [ subtractive 'I' "VX" subtractive 'X' "LC" subtractive 'C' "DM" ] |> Arb.fromGen Prop.forAll symbols <| fun (subtrahend, minuend) -> let originalRoman = String [| subtrahend; minuend |] let actual = Numeral.tryParseRoman originalRoman let roundTrippedRoman = actual |> Option.bind Numeral.toRoman let expected = opt { let! m = Numeral.tryParseRoman (string minuend) let! s = Numeral.tryParseRoman (string subtrahend) return m - s } expected =! actual Some originalRoman =! roundTrippedRoman
Like the previous property, this looks like a mouthful, but isn't too bad. I'll walk you through it.
Initially, ignore the OptionBuilder type and the opt value; we'll return to them shortly. The property itself starts by defining a function called subtractive, which has the type char -> string -> Gen<char * char>. The first argument is a symbol representing the subtrahend; that is: the number being subtracted. The next argument is a sequence of minuends; that is: numbers from which the subtrahend will be subtracted.
The subtractive function is implemented with a gen computation expression. It first uses a let! binding to define that a singular minuend is a random value drawn from minuends. As usual, Gen.elements is the workhorse: it defines a generator that randomly draws from a sequence of values, and because minuends is a string, and the string type implements char seq, it can be used with Gen.elements to define a generator of char values. While Gen.elements minuends returns a Gen<char> value, the use of a let! binding within a computation expression causes minuend to have the type char.
The second line of code in subtractive returns a tuple of two char values: the subtrahend first, and the minuend second. Normally, when subtracting with the arithmetic minus operator, you'd write a difference as minuend - subtrahend; the minuends comes first, followed by the subtrahend. The Roman subtractive notation, however, is written with the subtrahend before the minuend, which is the reason that the subtractive function returns the symbols in that order. It's easier to think about that way.
The subtractive function enables you to define generators of Roman numerals using subtractive notation. Since I can only be used before V and X, you can define a generator using subtractive 'I' "VX". This is a Gen<char * char> value. Likewise, you can define subtractive 'X' "LC" and subtractive 'C' "DM" and use Gen.oneOf to define a generator that randomly selects one of these generators, and uses the selected generator to produce a value. As always, the last step is to convert the generator into an Arbitrary with Arb.fromGen, so that symbols has the type Arbitrary<char * char>.
Equipped with an Arbitrary, you can again use Prop.forAll to express the desired property. First, originalRoman is created from the subtrahend and minuend. Due to the way symbols is defined, originalRoman will have values like "IV", "XC", and so on.
The property then proceeds to invoke tryParseRoman. It also uses the actual value to produce a round-tripped value. Not only should the parser correctly understand subtractive notation, but the integer-to-Roman conversion should also prefer this notation.
The last part of the property is the assertion. Here, you need opt, which is a computation builder for option values.
In the assertion, you need to calculate the expected value. Both minuend and subtrahend are char values; in order to find their corresponding decimal values, you'll need to call tryParseRoman. The problem is that tryParseRoman returns an int option. For example, tryParseRoman "I" returns Some 1, so you may need to subtract Some 1 from Some 5. The most readable way I've found is to use a computation expression. Using let! bindings, both m and s are int values, which you can easily subtract using the normal - operator.
expected and actual are both int option values, so can be compared using Unquote's must equal operator.
Finally, the property also asserts that the original value must be equal to the round-tripped value. If not, you could have an implementation that correctly parses "IV" as 4, but converts 4 to "IIII".
Limited repetition #
The previous property only ensures that subtractive notation is used in simple cases, like IX or CD. It doesn't verify that composite numerals are correctly written. As an example, the converter should convert 1893 to MDCCCXCIII, not MDCCCLXXXXIII. The second alternative is incorrect because it uses LXXXX to represent 90, instead of XC.
The underlying property is that any given symbol can only be repeated at most three times. A symbol can appear more than thrice in total, as demonstrated by the valid numeral MDCCCXCIII, in which C appears four times. For any group of repeated characters, however, a character must only be repeated once, twice, or thrice.
This also explain why the maximum Roman numeral is MMMCMXCIX, or 3,999.
In order to express this property in code, you first need a function to group characters. Here, I've chosen to reuse one of my own creation:
// seq<'a> -> 'a list list when 'a : equality let group xs = let folder x = function | [] -> [[x]] | (h::t)::ta when h = x -> (x::h::t)::ta | acc -> [x]::acc Seq.foldBack folder xs []
This function will, for example, group "MDCCCXCIII" like this:
> "MDCCCXCIII" |> group;; val it : char list list = [['M']; ['D']; ['C'; 'C'; 'C']; ['X']; ['C']; ['I'; 'I'; 'I']]
All you need to do is to find the length of all such sub-lists, and assert that the maximum is at most 3:
[<Property(QuietOnSuccess = true)>] let ``there can be no more than three identical symbols in a row`` () = Prop.forAll romanRange <| fun i -> let actual = Numeral.toRoman i test <@ actual |> Option.map (group >> (List.map List.length) >> List.max) |> Option.exists (fun x -> x <= 3) @>
Since actual is a string option, you need to express the assertion within the Maybe (Option) monad. First, you can use Option.map to map any value (should there be one) to find the maximum length of any repeated character group. This returns an int option.
Finally, you can pipe that int option into Option.exists, which will evaluate to false if there's no value, or if the boolean expression x <= 3 evaluates to false.
Input range #
At this point, you're almost done. The only remaining properties you'll need to specify is that the maximum value is 3,999, and the minimum value is 1. Negative numbers, or zero, are not allowed:
[<Property(QuietOnSuccess = true)>] let ``negative numbers and zero are not supported`` i = let i = -(abs i) let actual = Numeral.toRoman i test <@ Option.isNone actual @>
In this property, the function argument i can be any number, but calling abs ensures that it's positive (or zero), and the unary - operator then converts that positive value to a negative value.
Notice that the new i value shadows the input argument of the same name. This is a common trick when writing properties. It prevents me from accidentally using the input value provided by FsCheck. While the input argument is useful as a seed value, it isn't guaranteed to model the particular circumstances of this property. Here, you only care to assert what happens if the input is negative or zero. Specifically, you always want the return value to be None.
Likewise for too large input values:
[<Property(QuietOnSuccess = true)>] let ``numbers too big are not supported`` () = Gen.choose (4000, Int32.MaxValue) |> Arb.fromGen |> Prop.forAll <| fun i -> let actual = Numeral.toRoman i test <@ Option.isNone actual @>
Here, Gen.choose is used to define an Arbitrary<int> that only produces numbers between 4000 and Int32.MaxValue (including both boundary values).
This test is similar to the one that exercises negative values, so you could combine them to a single function if you'd like. I'll leave this as an exercise, though.
Implementation #
The interesting part of this exercise is, I think, how to define the properties. There are many ways you can implement the functions to pass all properties. Here's one of them:
open System let tryParseRoman candidate = let add x = Option.map ((+) x) let rec imp acc = function | 'I'::'X'::xs -> imp (acc |> add 9) xs | 'I'::'V'::xs -> imp (acc |> add 4) xs | 'I'::xs -> imp (acc |> add 1) xs | 'V'::xs -> imp (acc |> add 5) xs | 'X'::'C'::xs -> imp (acc |> add 90) xs | 'X'::'L'::xs -> imp (acc |> add 40) xs | 'X'::xs -> imp (acc |> add 10) xs | 'L'::xs -> imp (acc |> add 50) xs | 'C'::'M'::xs -> imp (acc |> add 900) xs | 'C'::'D'::xs -> imp (acc |> add 400) xs | 'C'::xs -> imp (acc |> add 100) xs | 'D'::xs -> imp (acc |> add 500) xs | 'M'::xs -> imp (acc |> add 1000) xs | [] -> acc | _ -> None candidate |> Seq.toList |> imp (Some 0) let toRoman i = let rec imp acc = function | x when x >= 1000 -> imp ('M':: acc) (x - 1000) | x when x >= 900 -> imp ('M'::'C'::acc) (x - 900) | x when x >= 500 -> imp ('D':: acc) (x - 500) | x when x >= 400 -> imp ('D'::'C'::acc) (x - 400) | x when x >= 100 -> imp ('C':: acc) (x - 100) | x when x >= 90 -> imp ('C'::'X'::acc) (x - 90) | x when x >= 50 -> imp ('L':: acc) (x - 50) | x when x >= 40 -> imp ('L'::'X'::acc) (x - 40) | x when x >= 10 -> imp ('X':: acc) (x - 10) | x when x >= 9 -> imp ('X'::'I'::acc) (x - 9) | x when x >= 5 -> imp ('V':: acc) (x - 5) | x when x >= 4 -> imp ('V'::'I'::acc) (x - 4) | x when x >= 1 -> imp ('I':: acc) (x - 1) | _ -> acc if 0 < i && i < 4000 then imp [] i |> List.rev |> List.toArray |> String |> Some else None
Both functions use tail-recursive inner imp functions in order to accumulate the appropriate answer.
One of the nice properties (that I didn't test for) of this implementation is that the tryParseRoman function is a Tolerant Reader. While toRoman would never create such a numeral, tryParseRoman correctly understands some alternative renderings:
> "MDCCCLXXXXIII" |> tryParseRoman;; val it : int option = Some 1893 > 1893 |> toRoman;; val it : String option = Some "MDCCCXCIII"
In other words, the implementation follows Postel's law. tryParseRoman is liberal in what it accepts, while toRoman is conservative in what it returns.
Summary #
Some problems look, at first glance, as obvious candidates for example-driven development. In my experience, this particularly happen when obvious examples abound. It's not difficult to come up with examples of Roman numerals, so it seems intuitive that you should just start writing some test cases with various examples. In my experience, though, that doesn't guarantee that you're led towards a good implementation.
The more a problem description is based on examples, the harder it can be to identify the underlying properties. Still, they're often there, once you start looking. As I've previously reported, using property-based test-driven development enables you to proceed in a more incremental fashion, because properties describe only parts of the desired solution.
If you're interested in learning more about Property-Based Testing, you can watch my introduction to Property-based Testing with F# Pluralsight course.
SUT Double
While it's possible to rely on Extract and Override for unit testing, Dependency Injection can make code and tests simpler and easier to maintain.
In object-oriented programming, many people still struggle with the intersection of design and testability. If a unit test is an automated test of a unit in isolation of its dependencies, then how do you isolate an object from its dependencies, most notably databases, web services, and the like?
One technique, described in The Art of Unit Testing, is called Extract and Override. The idea is that you write a class, but use a Template Method, Factory Method, or other sort of virtual class method to expose extensibility points that a unit test can use to achieve the desired isolation.
People sometimes ask me whether that isn't good enough, and why I advocate the (ostensibly) more complex technique of Dependency Injection?
It's easiest to understand the advantages and disadvantages if we have an example to discuss.
Example: Extract and Override #
Imagine that you're creating a reservation system for a restaurant. Clients (web sites, smart phone apps, etcetera) display a user interface where you can fill in details about your reservation: your name, email address, the number of guests, and the date of your reservation. When you submit your reservation request, the client POSTs a JSON document to a web service.
In this example, the web service is implemented by a Controller class, and the Post method handles the incoming request:
public int Capacity { get; } public IHttpActionResult Post(ReservationDto reservationDto) { DateTime requestedDate; if(!DateTime.TryParse(reservationDto.Date, out requestedDate)) return this.BadRequest("Invalid date."); var reservedSeats = this.ReadReservedSeats(requestedDate); if(this.Capacity < reservationDto.Quantity + reservedSeats) return this.StatusCode(HttpStatusCode.Forbidden); this.SaveReservation(requestedDate, reservationDto); return this.Ok(); }
The implementation is simple: It first attempts to validate the incoming document, and returns an error message if the document is invalid. Second, it reads the number of already reserved seats via a helper method, and rejects the request if the remaining capacity is insufficient. On the other hand, if the remaining capacity is sufficient, it saves the reservation and returns 200 OK.
The Post method relies on two helper methods that handles communication with the database:
public virtual int ReadReservedSeats(DateTime date) { const string sql = @" SELECT COALESCE(SUM([Quantity]), 0) FROM [dbo].[Reservations] WHERE YEAR([Date]) = YEAR(@Date) AND MONTH([Date]) = MONTH(@Date) AND DAY([Date]) = DAY(@Date)"; var connStr = ConfigurationManager.ConnectionStrings["booking"] .ConnectionString; using (var conn = new SqlConnection(connStr)) using (var cmd = new SqlCommand(sql, conn)) { cmd.Parameters.Add(new SqlParameter("@Date", date)); conn.Open(); return (int)cmd.ExecuteScalar(); } } public virtual void SaveReservation( DateTime dateTime, ReservationDto reservationDto) { const string sql = @" INSERT INTO [dbo].[Reservations] ([Date], [Name], [Email], [Quantity]) VALUES (@Date, @Name, @Email, @Quantity)"; var connStr = ConfigurationManager.ConnectionStrings["booking"] .ConnectionString; using (var conn = new SqlConnection(connStr)) using (var cmd = new SqlCommand(sql, conn)) { cmd.Parameters.Add( new SqlParameter("@Date", reservationDto.Date)); cmd.Parameters.Add( new SqlParameter("@Name", reservationDto.Name)); cmd.Parameters.Add( new SqlParameter("@Email", reservationDto.Email)); cmd.Parameters.Add( new SqlParameter("@Quantity", reservationDto.Quantity)); conn.Open(); cmd.ExecuteNonQuery(); } }
In this example, both helper methods are public, but they could have been protected without changing any conclusions; by being public, though, they're easier to override using Moq. The important detail is that both of these methods are overridable. In C#, you declare that with the virtual keyword; in Java, methods are overridable by default.
Both of these methods use elemental ADO.NET to communicate with the database. You can also use an ORM, or any other database access technique you prefer - it doesn't matter for this discussion. What matters is that the methods can be overridden by unit tests.
Here's a unit test of the happy path:
[Fact] public void PostReturnsCorrectResultAndHasCorrectStateOnAcceptableRequest() { var json = new ReservationDto { Date = "2016-05-31", Name = "Mark Seemann", Email = "mark@example.com", Quantity = 1 }; var sut = new Mock<ReservationsController> { CallBase = true }; sut .Setup(s => s.ReadReservedSeats(new DateTime(2016, 5, 31))) .Returns(0); sut .Setup(s => s.SaveReservation(new DateTime(2016, 5, 31), json)) .Verifiable(); var actual = sut.Object.Post(json); Assert.IsAssignableFrom<OkResult>(actual); sut.Verify(); }
This example uses the Extract and Override technique, but instead of creating a test-specific class that derives from ReservationsController, it uses Moq to create a dynamic Test Double for the System Under Test (SUT) - a SUT Double.
Since both ReadReservedSeats and SaveReservation are virtual, Moq can override them with test-specific behaviour. In this test, it defines the behaviour of ReadReservedSeats in such a way that it returns 0 when the input is a particular date.
While the test is simple, it has a single blemish. It ought to follow the Arrange Act Assert pattern, so it shouldn't have to configure the SaveReservation method before it calls the Post method. After all, the SaveReservation method is a Command, and you should use Mocks for Commands. In other words, the test ought to verify the interaction with the SaveReservation method in the Assert phase; not configure it in the Arrange phase.
Unfortunately, if you don't configure the SaveReservation method before calling Post, Moq will use the base implementation, which will attempt to interact with the database. The database isn't available in the unit test context, so without that override, the base method will throw an exception, causing the test to fail.
Still, that's a minor issue. In general, the test is easy to follow, and the design of the ReservationsController class itself is also straightforward. Are there any downsides?
Example: shared connection #
From a design perspective, the above version is fine, but you may find it inefficient that ReadReservedSeats and SaveReservation both open and close a connection to the database. Wouldn't it be more efficient if they could share a single connection?
If (by measuring) you decide that you'd like to refactor the ReservationsController class to use a shared connection, your first attempt might look like this:
public IHttpActionResult Post(ReservationDto reservationDto) { DateTime requestedDate; if (!DateTime.TryParse(reservationDto.Date, out requestedDate)) return this.BadRequest("Invalid date."); using (var conn = this.OpenDbConnection()) { var reservedSeats = this.ReadReservedSeats(conn, requestedDate); if (this.Capacity < reservationDto.Quantity + reservedSeats) return this.StatusCode(HttpStatusCode.Forbidden); this.SaveReservation(conn, requestedDate, reservationDto); return this.Ok(); } } public virtual SqlConnection OpenDbConnection() { var connStr = ConfigurationManager.ConnectionStrings["booking"] .ConnectionString; var conn = new SqlConnection(connStr); try { conn.Open(); } catch { conn.Dispose(); throw; } return conn; } public virtual int ReadReservedSeats(SqlConnection conn, DateTime date) { const string sql = @" SELECT COALESCE(SUM([Quantity]), 0) FROM [dbo].[Reservations] WHERE YEAR([Date]) = YEAR(@Date) AND MONTH([Date]) = MONTH(@Date) AND DAY([Date]) = DAY(@Date)"; using (var cmd = new SqlCommand(sql, conn)) { cmd.Parameters.Add(new SqlParameter("@Date", date)); return (int)cmd.ExecuteScalar(); } } public virtual void SaveReservation( SqlConnection conn, DateTime dateTime, ReservationDto reservationDto) { const string sql = @" INSERT INTO [dbo].[Reservations] ([Date], [Name], [Email], [Quantity]) VALUES (@Date, @Name, @Email, @Quantity)"; using (var cmd = new SqlCommand(sql, conn)) { cmd.Parameters.Add( new SqlParameter("@Date", reservationDto.Date)); cmd.Parameters.Add( new SqlParameter("@Name", reservationDto.Name)); cmd.Parameters.Add( new SqlParameter("@Email", reservationDto.Email)); cmd.Parameters.Add( new SqlParameter("@Quantity", reservationDto.Quantity)); cmd.ExecuteNonQuery(); } }
You've changed both ReadReservedSeats and SaveReservation to take an extra parameter: the connection to the database. That connection is created by the OpenDbConnection method, but you also have to make that method overridable, because otherwise, it'd attempt to connect to a database during unit testing, and thereby causing the tests to fail.
You can still unit test using the Extract and Overide technique, but the test becomes more complicated:
[Fact] public void PostReturnsCorrectResultAndHasCorrectStateOnAcceptableRequest() { var json = new ReservationDto { Date = "2016-05-31", Name = "Mark Seemann", Email = "mark@example.com", Quantity = 1 }; var sut = new Mock<ReservationsController> { CallBase = true }; sut.Setup(s => s.OpenDbConnection()).Returns((SqlConnection)null); sut .Setup(s => s.ReadReservedSeats( It.IsAny<SqlConnection>(), new DateTime(2016, 5, 31))) .Returns(0); sut .Setup(s => s.SaveReservation( It.IsAny<SqlConnection>(), new DateTime(2016, 5, 31), json)) .Verifiable(); var actual = sut.Object.Post(json); Assert.IsAssignableFrom<OkResult>(actual); sut.Verify(); }
Not only must you override ReadReservedSeats and SaveReservation, but you must also supply a Dummy Object for the connection object, as well as override OpenDbConnection. Still manageable, perhaps, but the design indisputably deteriorated.
You can summarise the flaw by a single design smell: Feature Envy. Both the ReadReservedSeats and the SaveReservation methods take an argument of the type SqlConnection. On the other hand, they don't use any instance members of the ReservationsController class that currently hosts them. These methods seem like they ought to belong to SqlConnection instead of ReservationsController. That's not possible, however, since SqlConnection is a class from the Base Class Library, but you can, instead, create a new Repository class.
Example: Repository #
A common design pattern is the Repository pattern, although the way it's commonly implemented today has diverged somewhat from the original description in PoEAA. Here, I'm going to apply it like people often do. You start by defining a new class:
public class SqlReservationsRepository : IDisposable { private readonly Lazy<SqlConnection> lazyConn; public SqlReservationsRepository() { this.lazyConn = new Lazy<SqlConnection>(this.OpenSqlConnection); } private SqlConnection OpenSqlConnection() { var connStr = ConfigurationManager.ConnectionStrings["booking"] .ConnectionString; var conn = new SqlConnection(connStr); try { conn.Open(); } catch { conn.Dispose(); throw; } return conn; } public virtual int ReadReservedSeats(DateTime date) { const string sql = @" SELECT COALESCE(SUM([Quantity]), 0) FROM [dbo].[Reservations] WHERE YEAR([Date]) = YEAR(@Date) AND MONTH([Date]) = MONTH(@Date) AND DAY([Date]) = DAY(@Date)"; using (var cmd = new SqlCommand(sql, this.lazyConn.Value)) { cmd.Parameters.Add(new SqlParameter("@Date", date)); return (int)cmd.ExecuteScalar(); } } public virtual void SaveReservation( DateTime dateTime, ReservationDto reservationDto) { const string sql = @" INSERT INTO [dbo].[Reservations] ([Date], [Name], [Email], [Quantity]) VALUES (@Date, @Name, @Email, @Quantity)"; using (var cmd = new SqlCommand(sql, this.lazyConn.Value)) { cmd.Parameters.Add( new SqlParameter("@Date", reservationDto.Date)); cmd.Parameters.Add( new SqlParameter("@Name", reservationDto.Name)); cmd.Parameters.Add( new SqlParameter("@Email", reservationDto.Email)); cmd.Parameters.Add( new SqlParameter("@Quantity", reservationDto.Quantity)); cmd.ExecuteNonQuery(); } } public void Dispose() { this.Dispose(true); GC.SuppressFinalize(this); } protected virtual void Dispose(bool disposing) { if (disposing) this.lazyConn.Value.Dispose(); } }
The new SqlReservationsRepository class contains the two ReadReservedSeats and SaveReservation methods, and because you've now moved them to a class that contains a shared database connection, the methods don't need the connection as a parameter.
The ReservationsController can use the new SqlReservationsRepository class to do its work, while keeping connection management efficient. In order to make it testable, however, you must make it overridable. The SqlReservationsRepository class' methods are already virtual, but that's not the class you're testing. The System Under Test is the ReservationsController class, and you have to make its use of SqlReservationsRepository overridable as well.
If you wish to avoid Dependency Injection, you can use a Factory Method:
public IHttpActionResult Post(ReservationDto reservationDto) { DateTime requestedDate; if (!DateTime.TryParse(reservationDto.Date, out requestedDate)) return this.BadRequest("Invalid date."); using (var repo = this.CreateRepository()) { var reservedSeats = repo.ReadReservedSeats(requestedDate); if (this.Capacity < reservationDto.Quantity + reservedSeats) return this.StatusCode(HttpStatusCode.Forbidden); repo.SaveReservation(requestedDate, reservationDto); return this.Ok(); } } public virtual SqlReservationsRepository CreateRepository() { return new SqlReservationsRepository(); }
The Factory Method in the above example is the CreateRepository method, which is virtual, and thereby overridable. You can override it in a unit test like this:
[Fact] public void PostReturnsCorrectResultAndHasCorrectStateOnAcceptableRequest() { var json = new ReservationDto { Date = "2016-05-31", Name = "Mark Seemann", Email = "mark@example.com", Quantity = 1 }; var repo = new Mock<SqlReservationsRepository>(); repo .Setup(r => r.ReadReservedSeats(new DateTime(2016, 5, 31))) .Returns(0); repo .Setup(r => r.SaveReservation(new DateTime(2016, 5, 31), json)) .Verifiable(); var sut = new Mock<ReservationsController> { CallBase = true }; sut.Setup(s => s.CreateRepository()).Returns(repo.Object); var actual = sut.Object.Post(json); Assert.IsAssignableFrom<OkResult>(actual); repo.Verify(); }
You'll notice that not only did the complexity increase of the System Under Test, but the test itself became more complicated as well. In the previous version, at least you only needed to create a single Mock<T>, but now you have to create two different test doubles and connect them. This is a typical example demonstrating the shortcomings of the Extract and Override technique. It doesn't scale well as complexity increases.
Example: Dependency Injection #
In 1994 we we were taught to favor object composition over class inheritance. You can do that, and still keep your code loosely coupled with Dependency Injection. Instead of relying on virtual methods, inject a polymorphic object into the class:
public class ReservationsController : ApiController { public ReservationsController(IReservationsRepository repository) { if (repository == null) throw new ArgumentNullException(nameof(repository)); this.Capacity = 12; this.Repository = repository; } public int Capacity { get; } public IReservationsRepository Repository { get; } public IHttpActionResult Post(ReservationDto reservationDto) { DateTime requestedDate; if (!DateTime.TryParse(reservationDto.Date, out requestedDate)) return this.BadRequest("Invalid date."); var reservedSeats = this.Repository.ReadReservedSeats(requestedDate); if (this.Capacity < reservationDto.Quantity + reservedSeats) return this.StatusCode(HttpStatusCode.Forbidden); this.Repository.SaveReservation(requestedDate, reservationDto); return this.Ok(); } }
In this version of ReservationsController, an IReservationsRepository object is injected into the object via the constructor and saved in a class field for later use. When the Post method executes, it calls Repository.ReadReservedSeats and Repository.SaveReservation without further ado.
The IReservationsRepository interface is defined like this:
public interface IReservationsRepository { int ReadReservedSeats(DateTime date); void SaveReservation(DateTime dateTime, ReservationDto reservationDto); }
Perhaps you're surprised to see that it merely defines the two ReadReservedSeats and SaveReservation methods, but makes no attempt at making the interface disposable.
Not only is IDisposable an implementation detail, but it also keeps the implementation of ReservationsController simple. Notice how it doesn't attempt to control the lifetime of the injected repository, which may or may not be disposable. In a few paragraphs, we'll return to this matter, but first, witness how the unit test became simpler as well:
[Fact] public void PostReturnsCorrectResultAndHasCorrectStateOnAcceptableRequest() { var json = new ReservationDto { Date = "2016-05-31", Name = "Mark Seemann", Email = "mark@example.com", Quantity = 1 }; var repo = new Mock<IReservationsRepository>(); repo .Setup(r => r.ReadReservedSeats(new DateTime(2016, 5, 31))) .Returns(0); var sut = new ReservationsController(repo.Object); var actual = sut.Post(json); Assert.IsAssignableFrom<OkResult>(actual); repo.Verify( r => r.SaveReservation(new DateTime(2016, 5, 31), json)); }
With this test, you can finally use the Arrange Act Assert structure, instead of having to configure the SaveReservation method call in the Arrange phase. This test arranges the Test Fixture by creating a Test Double for the IReservationsRepository interface and injecting it into the ReservationsController. You only need to configure the ReadReservedSeats method, because there's no default behaviour that you need to suppress.
You may be wondering about the potential memory leak when the SqlReservationsRepository is in use. After all, ReservationsController doesn't dispose of the injected repository.
You address this concern when you compose the dependency graph. This example uses ASP.NET Web API, which has an extensibility point for this exact purpose:
public class PureCompositionRoot : IHttpControllerActivator { public IHttpController Create( HttpRequestMessage request, HttpControllerDescriptor controllerDescriptor, Type controllerType) { if(controllerType == typeof(ReservationsController)) { var repo = new SqlReservationsRepository(); request.RegisterForDispose(repo); return new ReservationsController(repo); } throw new ArgumentException( "Unexpected controller type: " + controllerType, nameof(controllerType)); } }
The call to request.RegisterForDispose tells the ASP.NET Web API framework to dispose of the concrete repo object once it has handled the request and served the response.
Conclusion #
At first glance, it seems like the overall design would be easier if you make your SUT testable via inheritance, but once things become more complex, favouring composition over inheritance gives you more freedom to design according to well-known design patterns.
In case you want to dive into the details of the code presented here, it's available on GitHub. You can follow the progression of the code in the repository's commit history.
If you're interested in learning more about advanced unit testing techniques, you can watch my popular Pluralsight course.
TIE fighter FsCheck properties
Use the F# TIE fighter operator combination to eliminate a hard-to-name intermediary value.
A doctrine of Clean Code is the Boy Scout Rule: leave the code cleaner than you found it. Attempting to live by that rule, I'm always looking for ways to improve my code.
Writing properties with FsCheck is enjoyable, but I've been struggling with expressing ad-hoc Arbitraries in a clean style. Although I've already twice written about this, I've recently found another improvement.
Cleaner, but not clean enough #
Previously, I've described how you can use the backward pipe operator to avoid enclosing a multi-line expression in brackets:
[<Property(QuietOnSuccess = true)>] let ``Any live cell with > 3 live neighbors dies`` (cell : int * int) = let nc = Gen.elements [4..8] |> Arb.fromGen Prop.forAll nc <| fun neighborCount -> let liveNeighbors = cell |> findNeighbors |> shuffle |> Seq.take neighborCount |> Seq.toList let actual : State = calculateNextState (cell :: liveNeighbors |> shuffle |> set) cell Dead =! actual
This property uses the custom Arbitrary nc to express a property about Conway's Game of Life. The value nc is a Arbitrary<int>, which is a generator of integer values between 4 and 8 (both included).
There's a problem with that code, though: I don't care about nc; I care about the values it creates. That's the important value in my property, and this is the reason I reserved the descriptive name neighborCount for that role. The property cares about the neighbour count: for any neighbour count in the range 4-8, it should hold that blah blah blah, and so on...
Unfortunately, I was still left with the need to pass an Arbitrary<'a> to Prop.forAll, so I named it the best I could: nc (short for Neighbour Count). Following Clean Code's heuristics for short symbol scope, I considered it an acceptable name, albeit a bit cryptic.
TIE Fighters to the rescue! #
One day, I was looking at a property like the one above, and I thought: if you consider the expression Prop.forAll nc in isolation, you could also write it nc |> Prop.forAll. Does it work in this context? Yes it does:
[<Property(QuietOnSuccess = true)>] let ``Any live cell with > 3 live neighbors dies`` (cell : int * int) = let nc = Gen.elements [4..8] |> Arb.fromGen (nc |> Prop.forAll) <| fun neighborCount -> let liveNeighbors = cell |> findNeighbors |> shuffle |> Seq.take neighborCount |> Seq.toList let actual : State = calculateNextState (cell :: liveNeighbors |> shuffle |> set) cell Dead =! actual
This code compiles, and is equivalent to the first example. I deliberately put the expression nc |> Prop.forAll in brackets to be sure that I'd made a correct replacement. Pipes are left-associative, though, so in this case, the brackets are redundant:
[<Property(QuietOnSuccess = true)>] let ``Any live cell with > 3 live neighbors dies`` (cell : int * int) = let nc = Gen.elements [4..8] |> Arb.fromGen nc |> Prop.forAll <| fun neighborCount -> let liveNeighbors = cell |> findNeighbors |> shuffle |> Seq.take neighborCount |> Seq.toList let actual : State = calculateNextState (cell :: liveNeighbors |> shuffle |> set) cell Dead =! actual
This still works (and fails if the system under test has a defect).
Due to referential transparency, though, the value nc is equal to the expression Gen.elements [4..8] |> Arb.fromGen, so you can replace it:
[<Property(QuietOnSuccess = true)>] let ``Any live cell with > 3 live neighbors dies`` (cell : int * int) = Gen.elements [4..8] |> Arb.fromGen |> Prop.forAll <| fun neighborCount -> let liveNeighbors = cell |> findNeighbors |> shuffle |> Seq.take neighborCount |> Seq.toList let actual : State = calculateNextState (cell :: liveNeighbors |> shuffle |> set) cell Dead =! actual
In the above example, I've also slightly reformatted the expression, so that each expression composed with a forward pipe is on a separate line. That's not required, but I find it more readable.
Notice how Prop.forAll is now surrounded by pipes: |> Prop.forAll <|. This is humorously called TIE fighter infix.
Summary #
Sometimes, giving an intermediary value a descriptive name can improve code readability, but in other cases, such a value is only in the way. This was the case with the nc value in the first example above. Using TIE fighter infix notation enables you to get rid of a redundant, hard-to-name symbol. In my opinion, because nc didn't add any information to the code, I find the refactored version easier to read. I've left the code base cleaner than I found it.
CQS and server-generated Entity IDs
Is it a violation of Command Query Separation to update an ID property when saving an Entity to a database?
In my Encapsulation and SOLID course on Pluralsight, I explain how the elusive object-oriented quality encapsulation can be approximated by the actionable principles of Command Query Separation (CQS) and Postel's law.
One of the questions that invariably arise when people first learn about CQS is how to deal with (database) server-generated IDs. While I've already covered that question, I recently came upon a variation of the question:
"The Create method is a command, then if the object passed to this method have some changes in their property, does it violate any rule? I mean that we can always get the new id from the object itself, so we don't need to return another integer. Is it good or bad practice?"In this article, I'll attempt to answer this question.
Returning an ID by mutating input #
I interpret the question like this: an Entity (as described in DDD) can have a mutable Id property. A Id property with the newly created record's ID.
As an example, consider this User class:
public class User { public int Id { get; set; } public string FirstName { get; set; } public string LastName { get; set; } }
In order to create a new User in your database, you could define an API like this:
public interface IUserRepository { void Create(User user); }
An implementation of IUserRepository based on a relational database could perform an INSERT into the appropriate database, get the ID of the created record, and update the User object's Id property.
This test snippet demonstrates that behaviour:
var u = new User { FirstName = "Jane", LastName = "Doe" }; Assert.Equal(0, u.Id); repository.Create(u); Assert.NotEqual(0, u.Id);
When you create the u object, by not assigning a value to the Id property, it will have the default value of 0. Only after the Create method returns does Id hold a proper value.
Evaluation #
Does this design adhere to CQS? Yes, it does. The Create method doesn't return a value, but rather changes the state of the system. Not only does it create a record in your database, but it also changes the state of the u object. Nowhere does CQS state that an operation can't change more than a single part of the system.
Is it good design, then? I don't think that it is.
First, the design violates another part of encapsulation: protection of invariants. The invariants of any Entity is that is has an ID. It is the single defining feature of Entities that they have enduring and stable identities. When you change the identity of an Entity, it's no longer the same Entity. Yet this design allows exactly that:
u.Id = 42;
// ...
u.Id = 1337;
Second, such a design puts considerable trust in the implicit protocol between any client and the implementation of IUserRepository. Not only must the Create method save the User object in a data store, but it must also update the Id.
What happens, though, if you replay the method call:
repository.Create(new User { FirstName = "Ada", LastName = "Poe" }); repository.Create(new User { FirstName = "Ada", LastName = "Poe" });
This will result in duplicated entries, because the repository can't detect whether this is a replay, or simply two new users with the same name. You may find this example contrived, but in these days of cloud-based storage, it's common to apply retry strategies to clients.
If you use one of the alternatives I previously outlined, you will not have this problem.
Invariants #
Even if you don't care about replays or duplicates, you should still consider the invariants of Entities. As a minimum, you shouldn't be able to change the ID of an Entity. For a class like User, it'll also make client developers' job easier if it can provide some useful guarantees. This is part of Postel's law applied: be conservative in what you send. In this case, at least guarantee that no values will be null. Thus, a better (but by no means perfect) User class could be:
public class User { public User(int id) { if (id <= 0) throw new ArgumentOutOfRangeException( nameof(id), "The ID must be a (unique) positive value.");; this.Id = id; this.firstName = ""; this.lastName = ""; } public int Id { get; } private string firstName; public string FirstName { get { return this.firstName; } set { if (value == null) throw new ArgumentNullException(nameof(value)); this.firstName = value; } } private string lastName; public string LastName { get { return this.lastName; } set { if (value == null) throw new ArgumentNullException(nameof(value)); this.lastName = value; } } }
The constructor initialises all class fields, and ensure that Id is a positive integer. It can't ensure that the ID is unique, though - at least, not without querying the database, which would introduce other problems.
How do you create a new User object and save it in the database, then?
One option is this:
public interface IUserRepository { void Create(string firstName, string lastName); User Read(int id); void Update(User user); void Delete(int id); }
The Create method doesn't use the User class at all, because at this time, the Entity doesn't yet exist. It only exists once it has an ID, and in this scenario, this only happens once the database has stored it and assigned it an ID.
Other methods can still use the User class as either input or output, because once the Entity has an ID, its invariants are satisfied.
There are other problems with this design, though, so I still prefer the alternatives I originally sketched.
Conclusion #
Proper object-oriented design should, as a bare minimum, consider encapsulation. This includes protecting the invariants of objects. For Entities, it means that an Entity must always have a stable and enduring identity. Well-designed Entities guarantee that identity values are always present and immutable. This requirement tend to be at odds with most popular Object-Relational Mappers (ORM), which require that ID fields are externally assignable. In order to use ORMs, most programmers compromise on encapsulation, ending up with systems that are relational in nature, but far from object-oriented.
Async as surrogate IO
Haskell defines IO as an explicit context for handling effects. In F#, you can experiment with using Async as a surrogate.
As a reaction to my article on the relationship between Functional architecture and Ports and Adapters, Szymon Pobiega suggested on Twitter:
random idea: in c# “IO Int” translates to “Task<int>”. By consistently using this we can make type system help you even in c#If you know Haskell, you might object to that statement, but I don't think Szymon meant it literally. It's an idea worth exploring, I think.
Async as an IO surrogate #
Functions in Haskell are, by default, pure. Whenever you need to do something impure, like querying a database or sending an email, you need to explicitly do this in an impure context: IO. Don't let the name IO mislead you: it's not only for input and output, but for all impure actions. As an example, random number generation is impure as well, because a function that returns a new value every time isn't pure. (There are ways to model random number generation with pure functions, but let's forget about that for now.)
F#, on the other hand, doesn't make the distinction between pure and impure functions in its type system, so nothing 'translates' from IO in Haskell into an equivalent type in F#. Still, it's worth looking into Szymon's suggestion.
In 2013, C# finally caught up with F#'s support for asynchronous work-flows, and since then, many IO-bound .NET APIs have gotten asynchronous versions. The underlying API for C#'s async/await support is the Task Parallel Library, which isn't particularly constrained to IO-bound operations. The use of async/await, however, makes most sense for IO-bound operations. It improves resource utilisation because it can free up threads while waiting for IO-bound operations to complete.
It seems to be a frequent observation that once you start using asynchronous operations, they tend to be 'infectious'. A common rule of thumb is async all the way. You can easily call synchronous methods from asynchronous methods, but it's harder the other way around.
This is similar to the distinction between pure and impure functions. You can call pure functions from impure functions, but not the other way around. If you call an impure function from a 'pure' function in F#, the caller also automatically becomes impure. In Haskell, it's not even possible; you can call a pure function from an IO context, but not the other way around.
In F#, Async<'a> is more idiomatic than using Task<T>. While Async<'a> doesn't translate directly to Haskell's IO either, it's worth experimenting with the analogy.
Async I/O #
In the previous Haskell example, communication with the database was implemented by functions returning IO:
getReservedSeatsFromDB :: ConnectionString -> ZonedTime -> IO Int saveReservation :: ConnectionString -> Reservation -> IO ()
By applying the analogy of IO to Async<'a>, you can implement your F# data access module to return Async work-flows:
module SqlGateway = // string -> DateTimeOffset -> Async<int> let getReservedSeats connectionString (date : DateTimeOffset) = // ... // string -> Reservation -> Async<unit> let saveReservation connectionString (reservation : Reservation) = // ...
This makes it harder to compose the desired imp function, because you need to deal with both asynchronous work-flows and the Either monad. In Haskell, the building blocks are already there in the shape of the EitherT monad transformer. F# doesn't have monad transformers, but in the spirit of the previous article, you can define a computation expression that combines Either and Async:
type AsyncEitherBuilder () = // Async<Result<'a,'c>> * ('a -> Async<Result<'b,'c>>) // -> Async<Result<'b,'c>> member this.Bind(x, f) = async { let! x' = x match x' with | Success s -> return! f s | Failure f -> return Failure f } // 'a -> 'a member this.ReturnFrom x = x let asyncEither = AsyncEitherBuilder ()
This is the minimal implementation required for the following composition. A more complete implementation would also define a Return method, as well as other useful methods.
In the Haskell example, you saw how I had to use hoistEither and liftIO to 'extract' the values within the do block. This is necessary because sometimes you need to pull a value out of an Either value, and sometimes you need to get the value inside an IO context.
In an asyncEither expression, you need similar functions:
// Async<'a> -> Async<Result<'a,'b>> let liftAsync x = async { let! x' = x return Success x' } // 'a -> Async<'a> let asyncReturn x = async { return x }
I chose to name the first one liftAsync as a counterpart to Haskell's liftIO, since I'm using Async<'a> as a surrogate for IO. The other function I named asyncReturn because it returns a pure value wrapped in an asynchronous work-flow.
You can now compose the desired imp function from the above building blocks:
let imp candidate = asyncEither { let! r = Validate.reservation candidate |> asyncReturn let! i = SqlGateway.getReservedSeats connectionString r.Date |> liftAsync let! r = Capacity.check 10 i r |> asyncReturn return! SqlGateway.saveReservation connectionString r |> liftAsync }
Notice how, in contrast with the previous example, this expression uses let! and return! throughout, but that you have to compose each line with asyncReturn or liftAsync in order to pull out the appropriate values.
As an example, the original, unmodified Validate.reservation function has the type ReservationRendition -> Result<Reservation, Error>. Inside an asyncEither expression, however, all let!-bound values must be of the type Async<Result<'a, 'b>>, so you need to wrap the return value Result<Reservation, Error> in Async. That's what asyncReturn does. Since this expression is let!-bound, the r value has the type Reservation. It's 'double-unwrapped', if you will.
Likewise, SqlGateway.getReservedSeats returns a value of the type Async<int>, so you need to pipe it into liftAsync in order to turn it into an Async<Result<int, Error>>. When that value is let!-bound, then, i has the type int.
There's one change compared to the previous examples: the type of imp changed. It's now ReservationRendition -> Async<Result<unit, Error>>. While you could write an adapter function that executes this function synchronously, it's better to recall the mantra: async all the way!
Async Controller #
Instead of coercing the imp function to be synchronous, you can change the ReservationsController that uses it to be asynchronous as well:
// ReservationRendition -> Task<IHttpActionResult> member this.Post(rendition : ReservationRendition) = async { let! res = imp rendition match res with | Failure (ValidationError msg) -> return this.BadRequest msg | Failure CapacityExceeded -> return this.StatusCode HttpStatusCode.Forbidden | Success () -> return this.Ok () } |> Async.StartAsTask
Notice that the method body uses an async computation expression, instead of the new asyncEither. This is because, when returning a result, you finally need to explicitly deal with the error cases as well as the success case. Using a standard async expression enables you to let!-bind res to the result of invoking the asynchronous imp function, and pattern match against it.
Since ASP.NET Web API support asynchronous controllers, this works when you convert the async work-flow to a Task with Async.StartAsTask. Async all the way.
In order to get this to work, I had to overload the BadRequest, StatusCode, and Ok methods, because they are protected, and you can't use protected methods from within a closure. Here's the entire code for the Controller, including the above Post method for completeness sake:
type ReservationsController(imp) = inherit ApiController() member private this.BadRequest (msg : string) = base.BadRequest msg :> IHttpActionResult member private this.StatusCode statusCode = base.StatusCode statusCode :> IHttpActionResult member private this.Ok () = base.Ok () :> IHttpActionResult // ReservationRendition -> Task<IHttpActionResult> member this.Post(rendition : ReservationRendition) = async { let! res = imp rendition match res with | Failure (ValidationError msg) -> return this.BadRequest msg | Failure CapacityExceeded -> return this.StatusCode HttpStatusCode.Forbidden | Success () -> return this.Ok () } |> Async.StartAsTask
As you can see, apart from the overloaded methods, the Post method is all there is. It uses the injected imp function to perform the actual work, asynchronously translates the result into a proper HTTP response, and returns it as a Task<IHttpActionResult>.
Summary #
The initial idea was to experiment with using Async as a surrogate for IO. If you want to derive any value from this convention, you'll need to be disciplined and make all impure functions return Async<'a> - even those not IO-bound, but impure for other reasons (like random number generators).
I'm not sure I'm entirely convinced that this would be a useful strategy to adopt, but on the other hand, I'm not deterred either. At least, the outcome of this experiment demonstrates that you can combine asynchronous work-flows with the Either monad, and that the code is still manageable, with good separation of concerns. Since it is a good idea to access IO-bound resources using asynchronous work-flows, it's nice to know that they can be combined with sane error handling.
Comments
In my opinion, the true purpose of IO is to allow you to write pure effectful functions.
For example, Haskell's putStrLn is referentially transparent - i.e., [putStrLn "hello", putStrLn "hello"] has the same meaning as replicate 2 $ putStrLn "hello". This means you can treat a function that returns an IO the same way you'd treat any other function.
This is possible for two reasons: 1) IO actions are not run until they hit the "edge of the universe" where unsafePerformIO is called and 2) they don't memoize the result - if an IO action is executed twice, it will calculate two possibly distinct values.
The problem with Task is that it's not referentially transparent.
new[] { Task.Run(() => WriteLine("hello")), Task.Run(() => WriteLine("hello")) } is not the same as var t = Task.Run(() => WriteLine("hello")) + new [] { t, t }. Even if you use new Task() instead of Task.Run to delay its execution, you still wouldn't be able to factor out the task because it memoizes the result (in this case, void), so you'd still see "hello" only once on the screen.
Since Task cannot be used to model pure functions, I don't think it's a suitable replacement for IO.
Diogo, thank you for writing. The point about Tasks not being referentially transparent is well taken. You are, indeed, correct.
F# asynchronous work-flows work differently, though. Extending the experiment outlined in this article, you can attempt to 'translate' putStrLn to F#. Since the return type of putStrLn is IO (), you'll need a function that returns Async<unit>. Here's my naive attempt:
let aprintfn fmt = async { return printfn fmt }
Corresponding to your two Haskell examples, consider, then, these two F# values:
let explicitSeq = seq { yield aprintfn "hello"; yield aprintfn "hello" } let replicatedSeq = Seq.replicate 2 <| aprintfn "hello"
Both of these values have the type seq<Async<unit>>, and I'd expect them to be equivalent.
In Haskell, I used sequence to run your code in GHCI, turning [IO ()] into IO [()]. In F#, you can do something similar:
// seq<Async<'a>> -> Async<seq<'a>> let asequence xs = async { return Seq.map Async.RunSynchronously xs }
Using this function, you can evaluate both expressions:
> asequence explicitSeq |> Async.RunSynchronously;; hello hello val it : seq= seq [null; null] > asequence replicatedSeq |> Async.RunSynchronously;; hello hello val it : seq = seq [null; null]
As you can see, the result is the same in both cases, like you'd expect them to be.
I'm still not claiming that Async<'a> is a direct translation of Haskell's IO, but there are similarities. Admittedly, I've never considered whether F# asynchronous work-flows satisfy the monad laws, but it wouldn't surprise me if they did.
Ah, I see! It wasn't clear to me that async workflows were referentially transparent.
That being the case, it seems like they are, indeed, a good substitute for IO a.
It seems like the F# team got async right :) The C# team should take a page from their book...
I know what functional programmers generally and Mark specially mean when they say that function is (or is not) referentially transparent. Namely, a function is referentially transparent if and only if its behavior is unchanged when memozied.
I see how caching behavior varies among Haskell's IO, F#'s Async, and C#'s Task. However, the previous comments are the only times that I have anyone say that a type is (or is not) referentially transparent. In general, what does it mean for a type to be (or not to be) referentially transparent?
Tyson, language is imprecise. I take it that we meant that the behaviour that the type (here: an object) affords helps preserve referential transparency. It's fairly clear to me that it makes sense to say that Task<T> isn't referentially transparent, because of the behaviour shown by the above examples.
In C# or F#, we can't universally state that Async is always referentially transparent, because these languages allow impure actions to be passed to the type's members. Still, if we pass pure functions, the members are referentially transparent too.
In object-oriented programming, we often think of a type as an object: data with behaviour. In Haskell, you often have a type and an associated type class. In F#, some containers (like Async) have an associated computation expression builder; again, that's a set of functions.
If a set of associated functions or members are all referentially transparent, it makes sense to me to talk about the type as being referentially transparent, but all this is a view I've just retconned as I'm writing. You may be able to poke holes in it, so don't take it for more than it is.
Composition with an Either computation expression
A simple example of a composition using an F# computation expression.
In a previous article, you saw how to compose an application feature from functions:
let imp = Validate.reservation >> map (fun r -> SqlGateway.getReservedSeats connectionString r.Date, r) >> bind (fun (i, r) -> Capacity.check 10 i r) >> map (SqlGateway.saveReservation connectionString)
This is awkward because of the need to carry the result of Validate.reservation (r) on to Capacity.check, while also needing it for SqlGateway.getReservedSeats. In this article, you'll see a neater, alternative syntax.
Building blocks #
To recapitulate the context, you have this Result discriminated union from Scott Wlaschin's article about railway-oriented programming:
type Result<'TSuccess,'TFailure> = | Success of 'TSuccess | Failure of 'TFailure
A type with this structure is also sometimes known as Either, because the result can be either a success or a failure.
The above example also uses bind and map functions, which are implemented this way:
// ('a -> Result<'b,'c>) -> Result<'a,'c> -> Result<'b,'c> let bind f x = match x with | Success s -> f s | Failure f -> Failure f // ('a -> 'b) -> Result<'a,'c> -> Result<'b,'c> let map f x = match x with | Success s -> Success(f s) | Failure f -> Failure f
With these building blocks, it's trivial to implement a computation expression for the type:
type EitherBuilder () = member this.Bind(x, f) = bind f x member this.ReturnFrom x = x
You can add more members to EitherBuilder if you need support for more advanced syntax, but this is the minimal implementation required for the following example.
You also need a value of the EitherBuilder type:
let either = EitherBuilder ()
All this code is entirely generic, so you could put it (or a more comprehensive version) in a reusable library.
Composition using the Either computation expression #
Since this Either computation expression is for general-purpose use, you can also use it to compose the above imp function:
let imp candidate = either { let! r = Validate.reservation candidate let i = SqlGateway.getReservedSeats connectionString r.Date return! Capacity.check 10 i r |> map (SqlGateway.saveReservation connectionString) }
Notice how the r value can be used by both the SqlGateway.getReservedSeats and Capacity.check functions, because they are all within the same scope.
This example looks more like the Haskell composition in my previous article. It even looks cleaner in F#, but to be fair to Haskell, in F# you don't have to explicitly deal with IO as a monadic context, so that in itself makes things look simpler.
A variation using shadowing #
You may wonder why I chose to use return! with an expression using the map function. One reason was that I wanted to avoid having to bind a named value to the result of calling Capacity.check. Naming is always difficult, and while I find the names r and i acceptable because the scope is so small, I prefer composing functions without having to declare intermediary values.
Another reason was that I wanted to show you an example that resembles the previous Haskell example as closely as possible.
You may, however, find the following variation more readable. In order to use return instead of return!, you need to furnish a Return method on your computation builder:
type EitherBuilder () = member this.Bind(x, f) = bind f x member this.Return x = Success x
Nothing prevents you from adding both Return and ReturnFrom (as well as other) methods to the EitherBuilder class, but I'm showing you the minimal implementation required for the example to work.
Using this variation of EitherBuilder, you can alternatively write the imp function like this:
let imp candidate = either { let! r = Validate.reservation candidate let i = SqlGateway.getReservedSeats connectionString r.Date let! r = Capacity.check 10 i r return SqlGateway.saveReservation connectionString r }
In such an alternative, you can use a let!-bound value to capture the result of calling Capacity.check, but that obligates you to devise a name for the value. You'll often see names like r', but I always feel uneasy when I resort to such naming. Here, I circumvented the problem by shadowing the original r value with a new value also called r. This is possible, and even desirable, because once the reservation has been checked, you should use that new, checked value, and not the original value. That's what the original composition, above, does, so shadowing is both equivalent and appropriate.
Summary #
Computation expressions give you an opportunity to compose functions in a readable way. It's an alternative to using the underlying bind and map functions. Whether you prefer one over the other is partly subjective, but I tend to favour the underlying functions until they become inconvenient. The initial version of the imp function in this article is an example of map and bind becoming unwieldy, and where a computation expression affords a cleaner, more readable syntax.
The disadvantage of computation expressions is that you need to use return or return! to return a value. That forces you to write an extra line of code that isn't necessary with a simple bind or map.
Functional architecture is Ports and Adapters
Functional architecture tends to fall into a pit of success that looks a lot like Ports and Adapters.
In object-oriented architecture, we often struggle towards the ideal of the Ports and Adapters architecture, although we often call it something else: layered architecture, onion architecture, hexagonal architecture, and so on. The goal is to decouple the business logic from technical implementation details, so that we can vary each independently.
This creates value because it enables us to manoeuvre nimbly, responding to changes in business or technology.
Ports and Adapters #
The idea behind the Ports and Adapters architecture is that ports make up the boundaries of an application. A port is something that interacts with the outside world: user interfaces, message queues, databases, files, command-line prompts, etcetera. While the ports constitute the interface to the rest of the world, adapters translate between the ports and the application model.
The word adapter is aptly chosen, because the role of the Adapter design pattern is exactly to translate between two different interfaces.
You ought to arrive at some sort of variation of Ports and Adapters if you apply Dependency Injection, as I've previously attempted to explain.
The problem with this architecture, however, is that it seems to take a lot of explaining:
- My book about Dependency Injection is 500 pages long.
- Robert C. Martin's book about the SOLID principles, package and component design, and so on is 700 pages long.
- Domain-Driven Design is 500 pages long.
- and so on...
It's possible to implement a Ports and Adapters architecture with object-oriented programming, but it takes so much effort. Does it have to be that difficult?
Haskell as a learning aid #
Someone recently asked me: how do I know I'm being sufficiently Functional?
I was wondering that myself, so I decided to learn Haskell. Not that Haskell is the only Functional language out there, but it enforces purity in a way that neither F#, Clojure, nor Scala does. In Haskell, a function must be pure, unless its type indicates otherwise. This forces you to be deliberate in your design, and to separate pure functions from functions with (side) effects.
If you don't know Haskell, code with side effects can only happen inside of a particular 'context' called IO. It's a monadic type, but that's not the most important point. The point is that you can tell by a function's type whether or not it's pure. A function with the type ReservationRendition -> Either Error Reservation is pure, because IO appears nowhere in the type. On the other hand, a function with the type ConnectionString -> ZonedTime -> IO Int is impure because its return type is IO Int. This means that the return value is an integer, but that this integer originates from a context where it could change between function calls.
There's a fundamental distinction between a function that returns Int, and one that returns IO Int. Any function that returns Int is, in Haskell, referentially transparent. This means that you're guaranteed that the function will always return the same value given the same input. On the other hand, a function returning IO Int doesn't provide such a guarantee.
In Haskell programming, you should strive towards maximising the amount of pure functions you write, pushing the impure code to the edges of the system. A good Haskell program has a big core of pure functions, and a shell of IO code. Does that sound familiar?
It basically means that Haskell's type system enforces the Ports and Adapters architecture. The ports are all your IO code. The application's core is all your pure functions. The type system automatically creates a pit of success.
Haskell is a great learning aid, because it forces you to explicitly make the distinction between pure and impure functions. You can even use it as a verification step to figure out whether your F# code is 'sufficiently Functional'. F# is a Functional first language, but it also allows you to write object-oriented or imperative code. If you write your F# code in a Functional manner, though, it's easy to translate to Haskell. If your F# code is difficult to translate to Haskell, it's probably because it isn't Functional.
Here's an example.
Accepting reservations in F#, first attempt #
In my Test-Driven Development with F# Pluralsight course (a free, condensed version is also available), I demonstrate how to implement an HTTP API that accepts reservation requests for an on-line restaurant booking system. One of the steps when handling the reservation request is to check whether the restaurant has enough remaining capacity to accept the reservation. The function looks like this:
// int // -> (DateTimeOffset -> int) // -> Reservation // -> Result<Reservation,Error> let check capacity getReservedSeats reservation = let reservedSeats = getReservedSeats reservation.Date if capacity < reservation.Quantity + reservedSeats then Failure CapacityExceeded else Success reservation
As the comment suggests, the second argument, getReservedSeats, is a function of the type DateTimeOffset -> int. The check function calls this function to retrieve the number of already reserved seats on the requested date.
When unit testing, you can supply a pure function as a Stub; for example:
let getReservedSeats _ = 0 let actual = Capacity.check capacity getReservedSeats reservation
When finally composing the application, instead of using a pure function with a hard-coded return value, you can compose with an impure function that queries a database for the desired information:
let imp = Validate.reservation >> bind (Capacity.check 10 (SqlGateway.getReservedSeats connectionString)) >> map (SqlGateway.saveReservation connectionString)
Here, SqlGateway.getReservedSeats connectionString is a partially applied function, the type of which is DateTimeOffset -> int. In F#, you can't tell by its type that it's impure, but I know that this is the case because I wrote it. It queries a database, so isn't referentially transparent.
This works well in F#, where it's up to you whether a particular function is pure or impure. Since that imp function is composed in the application's Composition Root, the impure functions SqlGateway.getReservedSeats and SqlGateway.saveReservation are only pulled in at the edge of the system. The rest of the system is nicely protected against side-effects.
It feels Functional, but is it?
Feedback from Haskell #
In order to answer that question, I decided to re-implement the central parts of this application in Haskell. My first attempt to check the capacity was this direct translation:
checkCapacity :: Int
-> (ZonedTime -> Int)
-> Reservation
-> Either Error Reservation
checkCapacity capacity getReservedSeats reservation =
let reservedSeats = getReservedSeats $ date reservation
in if capacity < quantity reservation + reservedSeats
then Left CapacityExceeded
else Right reservation
This compiles, and at first glance seems promising. The type of the getReservedSeats function is ZonedTime -> Int. Since IO appears nowhere in this type, Haskell guarantees that it's pure.
On the other hand, when you need to implement the function to retrieve the number of reserved seats from a database, this function must, by its very nature, be impure, because the return value could change between two function calls. In order to enable that in Haskell, the function must have this type:
getReservedSeatsFromDB :: ConnectionString -> ZonedTime -> IO Int
While you can partially apply the first ConnectionString argument, the return value is IO Int, not Int.
A function with the type ZonedTime -> IO Int isn't the same as ZonedTime -> Int. Even when executing inside of an IO context, you can't convert ZonedTime -> IO Int to ZonedTime -> Int.
You can, on the other hand, call the impure function inside of an IO context, and extract the Int from the IO Int. That doesn't quite fit with the above checkCapacity function, so you'll need to reconsider the design. While it was 'Functional enough' for F#, it turns out that this design isn't really Functional.
If you consider the above checkCapacity function, though, you may wonder why it's necessary to pass in a function in order to determine the number of reserved seats. Why not simply pass in this number instead?
checkCapacity :: Int -> Int -> Reservation -> Either Error Reservation
checkCapacity capacity reservedSeats reservation =
if capacity < quantity reservation + reservedSeats
then Left CapacityExceeded
else Right reservation
That's much simpler. At the edge of the system, the application executes in an IO context, and that enables you to compose the pure and impure functions:
import Control.Monad.Trans (liftIO) import Control.Monad.Trans.Either (EitherT(..), hoistEither) postReservation :: ReservationRendition -> IO (HttpResult ()) postReservation candidate = fmap toHttpResult $ runEitherT $ do r <- hoistEither $ validateReservation candidate i <- liftIO $ getReservedSeatsFromDB connStr $ date r hoistEither $ checkCapacity 10 i r >>= liftIO . saveReservation connStr
(Complete source code is available here.)
Don't worry if you don't understand all the details of this composition. The highlights are these:
The postReservation function takes a ReservationRendition (think of it as a JSON document) as input, and returns an IO (HttpResult ()) as output. The use of IO informs you that this entire function is executing within the IO monad. In other words: it's impure. This shouldn't be surprising, since this is the edge of the system.
Furthermore, notice that the function liftIO is called twice. You don't have to understand exactly what it does, but it's necessary to use in order to 'pull out' a value from an IO type; for example pulling out the Int from an IO Int. This makes it clear where the pure code is, and where the impure code is: the liftIO function is applied to the functions getReservedSeatsFromDB and saveReservation. This tells you that these two functions are impure. By exclusion, the rest of the functions (validateReservation, checkCapacity, and toHttpResult) are pure.
It's interesting to observe how you can interleave pure and impure functions. If you squint, you can almost see how the data flows from the pure validateReservation function, to the impure getReservedSeatsFromDB function, and then both output values (r and i) are passed to the pure checkCapacity function, and finally to the impure saveReservation function. All of this happens within an (EitherT Error IO) () do block, so if any of these functions return Left, the function short-circuits right there and returns the resulting error. See e.g. Scott Wlaschin's excellent article on railway-oriented programming for an exceptional, lucid, clear, and visual introduction to the Either monad.
The value from this expression is composed with the built-in runEitherT function, and again with this pure function:
toHttpResult :: Either Error () -> HttpResult () toHttpResult (Left (ValidationError msg)) = BadRequest msg toHttpResult (Left CapacityExceeded) = StatusCode Forbidden toHttpResult (Right ()) = OK ()
The entire postReservation function is impure, and sits at the edge of the system, since it handles IO. The same is the case of the getReservedSeatsFromDB and saveReservation functions. I deliberately put the two database functions in the bottom of the below diagram, in order to make it look more familiar to readers used to looking at layered architecture diagrams. You can imagine that there's a cylinder-shaped figure below the circles, representing a database.
You can think of the validateReservation and toHttpResult functions as belonging to the application model. While pure functions, they translate between the external and internal representation of data. Finally, the checkCapacity function is part of the application's Domain Model, if you will.
Most of the design from my first F# attempt survived, apart from the Capacity.check function. Re-implementing the design in Haskell has taught me an important lesson that I can now go back and apply to my F# code.
Accepting reservations in F#, even more Functionally #
Since the required change is so little, it's easy to apply the lesson learned from Haskell to the F# code base. The culprit was the Capacity.check function, which ought to instead be implemented like this:
let check capacity reservedSeats reservation = if capacity < reservation.Quantity + reservedSeats then Failure CapacityExceeded else Success reservation
This simplifies the implementation, but makes the composition slightly more involved:
let imp = Validate.reservation >> map (fun r -> SqlGateway.getReservedSeats connectionString r.Date, r) >> bind (fun (i, r) -> Capacity.check 10 i r) >> map (SqlGateway.saveReservation connectionString)
This almost looks more complicated than the Haskell function. Haskell has the advantage that you can automatically use any type that implements the Monad typeclass inside of a do block, and since (EitherT Error IO) () is a Monad instance, the do syntax is available for free.
You could do something similar in F#, but then you'd have to implement a custom computation expression builder for the Result type. Perhaps I'll do this in a later blog post...
Summary #
Good Functional design is equivalent to the Ports and Adapters architecture. If you use Haskell as a yardstick for 'ideal' Functional architecture, you'll see how its explicit distinction between pure and impure functions creates a pit of success. Unless you write your entire application to execute within the IO monad, Haskell will automatically enforce the distinction, and push all communication with the external world to the edges of the system.
Some Functional languages, like F#, don't explicitly enforce this distinction. Still, in F#, it's easy to informally make the distinction and compose applications with impure functions pushed to the edges of the system. While this isn't enforced by the type system, it still feels natural.
Comments
Great post as always. However, I saw that you are using IO monad. I wonder if you shouldn't use IO action instead? To be honest, I'm not proficient in Haskell. But according to that post they are encouraging to use IO action rather than IO monad if we are not talking about monadic properties.
Arkadiusz, thank you for writing. It's the first time I see that post, but if I understand the argument correctly, it argues against the (over)use of the monad terminology when talking about IO actions that aren't intrinsically monadic in nature. As far as I can tell, it doesn't argue against the use of the IO type in Haskell.
It seems a reasonable point that it's unhelpful to newcomers to throw them straight into a discussion about monads, when all they want to see is a hello world example.
As some of the comments to that post point out, however, you'll soon need to compose IO actions in Haskell, and it's the monadic nature of the IO type that enables the use of do blocks. (..or, if you want to be more specific, it seems that sometimes, all you need is a Functor. It hardly helps to talk about the IO functor instead of the IO monad, though...)
I don't claim to be a Haskell expert, so there may be more subtle nuances in that article that simply went over my head. Still, within my current understanding, the discussion in this particular post of mine does relate to the IO monad.
When I wrote the article, I chose my words with some caution. Notice that when I introduce the IO type my article, I mostly talk about it as a 'context'.
When it comes to the discussion about the postReservation function, however, I pull out the stops and increasingly use the word monad. The reason is that this composition wouldn't be possible without the monadic properties of IO. Most of this function executes within a monadic 'stack': EitherT Error IO. EitherT is a monad transformer, and in order to be useful (e.g. composable in do blocks), the type it transforms must be a monad as well (or perhaps, as hinted above, a functor would be sufficient).
I agree with Justin Le's article that overuse of monad is likely counterproductive. On the other hand, one can also fall into the opposite side. There are some educators who seem to avoid the word at any cost. That's not my policy. I try to use as precise language as possible. That means that I'll use monad when I talk about monads, and I'll avoid it when it isn't necessary. That's my goal, but I may not have achieved it. Do you think I used it incorrectly here?
Mark, thank you for your comprehensive explanation. I was rather just asking you about opinion on this topic. I didn't try to claim that you had used it incorrectly. Now, I see that as you wrote you are first talking about IO context and then about IO monad when it comes to use monadic properties.
Hello, thanks so much for your website and pluralsight library -- it's been a tremendous help to me getting started with F# and functional programming (and TDD to a degree). I'm having some problems with the big picture question of how to structure applications, and this post is leaving me even more confused.
I can't understand how some of the original code, and especially the changes made in this post could possibly scale to a larger or more complicated application. Many real-life domain operations are going to need to query some data, make decisions based on that data, map that data into different domain objects, persist something, and so on.
It seems like the end result is that we end up copying much of the meaningful internal logic of the domain out to this boundary area in the Web API project. In this code we've extracted:
- The logic of only checking existing reservations on the same day as the day on the request
- The logic of not saving a new reservation if the request failed the capacity check
Does that logic not properly belong 'behind the wall' inside the domain? I would have expected to see a simple 'makeReserveration r' function as the public face to the domain. Suppose the restaurant also had a desktop/kiosk application in their building to manage their reservation system. Wouldn't we now be forced to duplicate this internal logic in the Composition Root of that application?
What I instinctively want to gravitate towards is something like this:
let internal makeReservation' getReservedSeats saveReservation check r = (getReservedSeats r.Date, r) |> bind (fun (i, r) -> check 10 i r) |> map saveReservation let makeReservation = makeReservation' SqlGateway.getReservedSeats SqlGateway.saveReservation Capacity.check
With this, the SqlGateway code needs be visible to the reservation code, putting it at the center of the app not on the edges. However, are there any practical disadvantages to this if the code is disciplined to only use it within this sort of dependency injection? What problems would I be likely to run into?
At least as a beginner, this seems to me to have a couple advantages:
- From the perspective of the unit tests (which test makeReservation') it is a pure function, as you noted above. Testability is no problem here.
- Code at the UI/Presentation level is nice and tight -- just make sure you can successfully transform the JSON request into a domain request, then process the command result.
But most notable to me, dependencies are 'injected' right where they were needed. Imagine a much more complicated aggregate operation like 'performOvernightCloseout' that needs to do many things like read cash register totals and send them to accounting, send timeclock info to HR, write order statistics to a reporting database, check inventory and create new orders for the following day. Of course we'd break this down into a hierarchy of functions, with top level orchestration calling down into increasingly specific functions, no function individually being too complicated.
The original demo application would have to approach this scenario by passing in a huge number of functions to the operation root, which then parcels them out level by level where they are needed. This would seem very brittle since any change in a low level function would require changes to the parameter list all the way up.
With newer code shown in this post, it would seem impossible. There might be a handful of small functions that can be in the central library, but the bulk of the logic is going to need to be extracted into a now-500-line imp method in the Web API project, or whatever service is launching this operation.
John, thank you for writing. I'm not sure I can answer everything to your satisfaction in single response, but we can always view this as a beginning of longer interchange. Some of the doubts you express are ambiguous, so I'm not sure how to interpret them, but I'll do my best to be as explicit as possible.
In the beginning of your comment, you write:
"Many real-life domain operations are going to need to query some data, make decisions based on that data, map that data into different domain objects, persist something, and so on."This is exactly the reason why I like the reservation request example so much, because it does all of that. It queries data, because it reads the number of already reserved seats from a database. It makes decisions based on that data, because it decides whether or not it can accept the reservation based on the number of remaining seats. It maps from an external JSON (or XML) data format to a Reservation record type (belonging to the Domain Model, if you will); it also maps back to an HTTP response. Finally, the example also saves the reservation to the database, if it was accepted.
When you say that we've extracted the logic, it's true, but I'm not sure what you mean by "behind the wall", but let's look at each of your bullet points in turn.
Querying existing reservations
You write that I've extracted the "logic of only checking existing reservations on the same day as the day on the request". It's a simple database query. In F#, one implementation may look like this:
// In module SqlGateway: let getReservedSeats connectionString (date : DateTimeOffset) = let min = DateTimeOffset(date.Date , date.Offset) let max = DateTimeOffset(date.Date.AddDays 1., date.Offset) let sql = " SELECT ISNULL(SUM(Quantity), 0) FROM [dbo].[Reservations] WHERE @min <= Date AND Date < @max" use conn = new SqlConnection(connectionString) use cmd = new SqlCommand(sql, conn) cmd.Parameters.AddWithValue("@min", min) |> ignore cmd.Parameters.AddWithValue("@max", max) |> ignore conn.Open() cmd.ExecuteScalar() :?> int
In my opinion, there's no logic in this function, but this is an example where terminology can be ambiguous. To me, however, logic implies that decisions are being made. That's not the case here. This getReservedSeats function has a cyclomatic complexity of 1.
This function is an Adapter: it adapts the ADO.NET SDK (the port to SQL Server) to something the Domain Model can use. In this case, the answer is a simple integer.
To save, or not to save
You also write that I've extracted out the "logic of not saving a new reservation if the request failed the capacity check". Yes, that logic is extracted out to the check function above. Since that's a pure function, it's part of the Domain Model.
The function that saves the reservation in the database, again, contains no logic (as I interpret the word logic):
// In module SqlGateway: let saveReservation connectionString (reservation : Reservation) = let sql = " INSERT INTO Reservations(Date, Name, Email, Quantity) VALUES(@Date, @Name, @Email, @Quantity)" use conn = new SqlConnection(connectionString) use cmd = new SqlCommand(sql, conn) cmd.Parameters.AddWithValue("@Date", reservation.Date) |> ignore cmd.Parameters.AddWithValue("@Name", reservation.Name) |> ignore cmd.Parameters.AddWithValue("@Email", reservation.Email) |> ignore cmd.Parameters.AddWithValue("@Quantity", reservation.Quantity) |> ignore conn.Open() cmd.ExecuteNonQuery() |> ignore
Due to the way the Either monad works, however, this function is only going to be called if all previous functions returned Success (or Right in Haskell). That logic is an entirely reusable abstraction. It's also part of the glue that composes functions together.
Reuse of composition
Further down, you ask:
"Suppose the restaurant also had a desktop/kiosk application in their building to manage their reservation system. Wouldn't we now be forced to duplicate this internal logic in the Composition Root of that application?"That's an excellent question! I'm glad you asked.
A desktop or kiosk application is noticeably different from a web application. It may have a different application flow, and it certainly exposes a different interface to the outside world. Instead of handling incoming HTTP requests, and translating results to HTTP responses, it'll need to respond to click events and transform results to on-screen UI events. This means that validation may be different, and that you don't need to map results back to HTTP response values. Again, this may be clearer in the Haskell implementation, where the toHttpResult function would obviously be superfluous. Additionally, a UI may present a native date and time picker control, which can directly produce strongly typed date and time values, negating the need for validation.
Additionally, a kiosk may need to be able to work in off-line, or occasionally connected, mode, so perhaps you'd prefer a different data store implementation. Instead of connecting directly to a database, such an application would need to write to a message queue, and read from an off-line snapshot of reservations.
That's a sufficiently different application that it warrants creating applications from fine-grained building blocks instead of course-grained building blocks. Perhaps the only part you'll end up reusing is the core business logic contained in the check[Capacity] function. Composition Roots aren't reusable.
(As an aside, you may think that if the check[Capacity] function is the only reusable part, then what's the point? This function is only a couple of lines long, so why even bother? That'd be a fair concern, if it wasn't for the context that all of this is only example code. Think of this example as a stand-in for much more complex business logic. Perhaps the restaurant will allow a certain percentage of over-booking, because some people never show up. Perhaps it will allow different over-booking percentages on different (week) days. Perhaps it'll need to handle reservations on per-table basis, instead of per-seat. Perhaps it'll need to handle multiple seatings on the same day. There are lots of variations you can throw into this business domain that will make it much more complex, and worthy of reuse.)
Brittleness of composition
Finally, you refer to a more realistic scenario, called performOvernightCloseout, and ask if a fine-grained composition wouldn't be brittle? In my experience, it wouldn't be more brittle than any alternatives I've identified. Whether you 'inject' functions into other functions, or you compose them in the Composition Root, doesn't change the underlying forces that act on your code. If you make substantial changes to the dependencies involved, it will break your code, and you'll need to address that. This is true for any manual composition, including Pure DI. The only way to avoid compilation errors when you redefine your dependency graphs is to use a DI Container, but that only makes the feedback loop worse, because it'd change a compile-time error into a run-time error.
This is a long answer, and even so, I'm not sure I've sufficiently addressed all of your concerns. If not, please write again for clarifications.
I believe there was a bit of miscommunication. When I talked about the 'logic of only checking existing reservations on the same day', I wasn't talking about the implementation of the SqlGateway. I was talking about the 'imp' function:
let imp = Validate.reservation >> map (fun r -> SqlGateway.getReservedSeats connectionString r.Date, r) >> bind (fun (i, r) -> Capacity.check 10 i r) >> map (SqlGateway.saveReservation connectionString)
This implements the logic of 'Validation comes first. If that checks out, then get the number of reserved seats from the data store. Use the date on the request for that. Feed this into the capacity check routine and only if everything checks out should we persist the reservation.
That's all internal logic that in my mind ought to be encapsulated within the domain. That's what I meant by 'behind the wall'. There should simply be a function 'makeReservation' with the signature ReservationRequest -> ReservationResponse. That's it. Clients should not be worrying themselves with these extra details. That 'imp' function just seems deeply inappropriate as something that lives externally to the reservations module, and that must be supplied individually by each client accessing the module.
I guess I'm rejecting the idea of Onion/Hexagonal architecture, or at least the idea that all I/O always belongs at the outside. To me, a reservations module must depend on a reservations data store, because if you don't persist information to the db, you haven't actually made the reservation. Of course, I still want to depend on interfaces, not implementations (perhaps naming the reference ReservationDataStore.saveReservation). The data store IS a separate layer and should have a well defined interface.
I just don't understand why anyone would find this design more desirable than the straightforward UI -> Logic -> Storage chain of dependencies. Clearly there's some sort of appeal -- lots of smart people show excitement for it. But it's a bit mystifying to me.
John, thank you for writing. It's OK to reject the concept of ports and adapters; it's not a one-size-fits-all architecture. While I often use it, I don't always use it. One of my most successful customer engagements involved building a set of REST APIs on top of an existing IT infrastructure. All the business logic was already implemented elsewhere, so I deliberately chose a simpler, more coupled architecture, because there'd be no benefit from a full ports and adapters architecture. The code bases involved still run in production four years later, even though they're constantly being changed and enhanced.
If, on the other hand, you choose to adopt the ports and adapters architecture, you'll need to do it right so that you can harvest the benefits.
A Façade function with the type ReservationRequest -> ReservationResponse only makes sense in an HTTP request/response scenario, because both the ReservationRequest and ReservationResponse types are specific to the web context. You can't reuse such a function in a different setting.
You still have such a function, because that's the function that handles the web request, but you'll need to compose it from smaller components, unless you want to write a Transaction Script within that function (which is also sometimes OK).
Granted, there's some 'logic' involved in this composition, in the sense that it might be possible to create a faulty composition, but that logic has to go somewhere.
Hi Mark, thanks for this post and the discussion.
I feel that I share some of the concerns that John is bringing up. The postReservation function is pretty simple now, but eventually we might want to make external calls based on some conditional logic from the domain model.
Let's say there is a new feature request. The restaurant has struck a deal with the local caravan dealership, allowing them to rent a caravan to park outside the restaurant in order to increase the seating capacity for one evening. Of course, sometimes there are no caravans available, so we'll need to query the caravan database to see if there is a big enough caravan available that evening:
findCaravan :: Int -> ZoneTime -> IO (Maybe Caravan)
findCaravan minimumCapacity date = ... -- database stuff
How would we integrate this? We could put a call to findCaravan inside the IO context of postReservation, but since we only want to query the caravan database if the first capacity check failed, we'll need to add branching logic to postReservation. We want to have tests for this logic, but since we are coupled to the IO context, we are in trouble.
The original issue was that the F# version of checkCapacity could be made impure by injecting the impure getReservedSeats function to it, which is something that Haskell disallows. What if we changed the Haskell version of checkCapacity to the following instead?
checkCapacity :: Monad m => Int
-> (ZonedTime -> m Int)
-> Reservation
-> EitherT Error m Reservation
checkCapacity capacity getReservedSeats reservation = EitherT $ do
reservedSeats <- getReservedSeats $ date reservation
if capacity < quantity reservation + reservedSeats
then return $ Left CapacityExceeded
else return $ Right reservation
Since we know that getReservedSeats will be effectful, we declare it so in its type signature. We change postReservation to become:
postReservation :: ReservationRendition -> IO (HttpResult ()) postReservation candidate = fmap toHttpResult $ runEitherT $ hoistEither $ validateReservation candidate >>= checkCapacity 10 (getReservedSeatsFromDB connStr) >>= liftIO . saveReservation connStr
(Or something like that. Monad transformers are the current step on my Haskell learning journey, so I'm not sure this will typecheck.)
This way, we can once more inject an IO typed function as the getReservedSeats argument of checkCapacity like we did in F#. If we want to test it, we can create a stub database call by using the monad instance of Identity, for example:
stubGetReservedSeats :: ZonedTime -> Identity Int
stubGetReservedSeats date = return 8
The new feature request also becomes easy to implement. Just pass findCaravan as a new argument to checkCapacity, and make the type of that argument to be ZonedTime -> m (Maybe Caravan). The checkCapacity function can now do the initial capacity check, and if that fails, call findCaravan to see if it is still possible to allow the booking by renting a suitable caravan. The branching logic that is needed is now inside checkCapacity and can be tested outside of the IO context.
I don't know if this would work well in practice, or if it would cause more trouble than what it's worth. What do you think?
Martin, thank you for writing. Your question prompted me to write a new article. It takes a slightly different approach to the one you suggest.
I haven't tried defining checkCapacity in the way you suggest, but that may be an exercise for another day. In general, though, I'm hesitant to introduce constraints not required by the implementation. In this case, I think it'd be a fair question to ask why the getReservedSeats argument has to have the type ZonedTime -> m Int? Nothing in its implementation seems to indicate the need for that, so it looks like a leaky abstraction to me.
I'd like to follow up my twitter comments here
Very interesting article! I'm sorry that some Haskell details are quite difficult for me at the moment and the F# code looks a bit too simplified or maybe just incomplete
So I'd prefer your approach here : it better translates Haskell to F# but I still have to study its implications. Anyway the concrete benefits of Async are evident, like the ones of F# computation expressions (to wrap exceptions etc...)
On the other hand I fail to see the benefit of purity vs other IoC patterns. Besides async and exceptions wrapping, purity seems an unnecessarily strict enforcement of Haskell (isn't it?): original F# was testable in isolation
What's missing if I only mock/stub/inject the dependency? When you say
"ability to reason is eroded"and
"it breaks encapsulation"about DI, do you mean something related to this topic?
Thank you so much for your interesting insights!
Why do you think that the only goal of pure functions is testability?My bad, actually I don't know. I assume that purity's goal is to avoid side effects. It is very clear that this can be useful when I have a hidden side effect in the function, but it is less obvious (to me) what it does mean when the side effect is as evident as an argument (?).
Anyway, I'll follow your implicit suggestion that with a pure function I can reach a more important goal than testability. Ok, practical example. Today I spent one hour to fix a bug that I introduced with a previous enhancement. My goal is to avoid this next time, by learning a better programming style from your article, but first I need to understand what went wrong. I can spot 3 problems:
- 1) I didn't see the bug when I wrote the code changes
- 2) I didn't go through that part of my app when I tested the upgrade
- 3) I needed a whole hour to fix that stupid bug, because I could not immediately find the point where it was
Now unfortunately I struggle to interpret Haskell's details and I can't find enough F# to recover the whole meaning, but the main point I get is that a part of code (that could be bugged because the ability to reason is eroded there) has been moved from a finally 'pure' function to a new 'impure' one, that is introduced by a special 'liftIO' keyword (sorry for my poor recap)
Finally my question is, but what about that all the possible bugs (often at the interface between persistence and model layers) that have simply been migrated around this critical IO land? are they going to counterbalance the above said benefits? where is exactly the net, overall improvement?
First, I should make it specific that when I discuss object-oriented code in general, and encapsulation specifically, I tend to draw heavily on the work of Bertrand Meyer in Object-Oriented Software Construction. Some people may regard Alan Kay as the original inventor of object-orientation, but C# and Java's interpretation of object-oriented programming seems closer to Meyer's than to Kay's.
To Meyer, the most important motivation for object-orientation was reuse. He wanted to be able to reuse software that other people had written, instead of, as had been the norm until then, importing other people's source code into new code bases. I'm sure you can see how that goal was important.
In order to make binary libraries reusable, Meyer wanted to establish some techniques that would enable developers to (re)use the code without having access to the source code. His vision was that every object should come with a set of invariants, pre-, and post-conditions. He even invented a programming language, Eiffel, that included the ability to define such rules as programmable and enforceable checks. Furthermore, such rules could be exported as automatic documentation. This concept, which Meyer called contracts, is what we now know as encapsulation (I have a Pluralsight course that explains this in more details).
While the idea of formal contracts seemed like a good idea, neither Java nor C# have them. In C# and Java, you can still implement invariants with guard clauses, assertions, and so on, but there's no formal, externally visible way to communicate such invariants.
In other words, in modern object-oriented languages, you can't reason about the code from its API alone. Meyer's goal was that you should be able to reuse an object that someone else had written, without reading the source code. In a nutshell: if you have to read the source code, it means that encapsulation is broken.
One of the other concepts that Meyer introduced was Command-Query Separation (CQS). To Meyer, it was important to be able to distinguish between operations that had side-effects, and operations that didn't.
In my experience, a big category of defects are caused by unintended side-effects. One developer doesn't realise that the method call (s)he invokes has a side-effect that another developer put there. Languages like C# and Java have few constraints, so any method could potentially hide a side-effect. The only way you can know is by reading the code.
What if, then, there was a programming language where you could tell, at a glance, whether an operation had a side-effect? Such programming languages exist. Haskell is one of them. In Haskell, all functions are pure by default. This means that they are deterministic and have no side-effects. If you want to write an impure function, you must explicitly declare that in the function's type (which is done with the IO type).
In Haskell, you don't have to read the source code in order to know whether a function is impure or not. You can simply look at the type!
This is one of the many benefits of pure functions: we know, by definition, that they have no side-effects. This property holds not only for the function itself, but for all functions that that pure function calls. This follows from the definition of purity. A pure function can call other pure functions, but it can't call impure functions, because, if it did, it would become impure itself. Haskell enforces this rule, so when you're looking at a pure function, you know that that entire subgraph of your code is pure. Otherwise, it wouldn't compile.
Not only does that relationship force us to push impure functions to the boundary of applications, as described in the present article. It also makes functions intrinsically testable. There are other advantages of pure functions besides these, such as easy composability, but in general, pure functions are attractive because it becomes easier to figure out what a function does without having to understand all the details. In other words, pure functions are better abstractions.
To be clear: functional programming doesn't protect you from defects, but in my experience, it helps you to avoid entire classes of bugs. I still write automated tests of my F# code, but I write fewer tests than when I wrote C# code. Despite that, I also seem to be producing fewer defects, and I rarely need to debug the functions.
What's the overall net improvement? I don't know. How would you measure that?
Consider a logic that, depending on a condition, performs one IO operation or another. In Java (I don’t know F# or Haskell):
int foo(boolean condition, IntSupplier io1, IntSupplier io2) {
if (condition) {
return io1.getAsInt();
} else {
return io2.getAsInt();
}
}
Would you recommend pre-fetching the two values prior to calling the foo function even though only one would ultimately be required?
Abel, thank you for writing. There's a discussion closely related to that question in the remarks sections of my dependency rejection article. It's the remark from 2017-02-18 19:54 UTC (I really ought to get around to add permalinks to comments...).
Additionally, there's a discussion on this page that lead to a new article that goes into some further details.
(This is rather a delayed reply, but...) @Martin Rykfors, you mentioned monad transformers as an option. A few months before your comment I recorded a livecoded session of applying MTL-style monad transformers to this code.
I'm not sure if you're interested (it's nearly an hour of me talking to myself), but just in case... 🙂
Ad-hoc Arbitraries - now with pipes
Slightly improved syntax for defining ad-hoc, in-line Arbitraries for FsCheck.Xunit.
Last year, I described how to define and use ad-hoc, in-line Arbitraries with FsCheck.Xunit. The final example code looked like this:
[<Property(QuietOnSuccess = true)>] let ``Any live cell with > 3 live neighbors dies`` (cell : int * int) = let nc = Gen.elements [4..8] |> Arb.fromGen Prop.forAll nc (fun neighborCount -> let liveNeighbors = cell |> findNeighbors |> shuffle |> Seq.take neighborCount |> Seq.toList let actual : State = calculateNextState (cell :: liveNeighbors |> shuffle |> set) cell Dead =! actual)
There's one tiny problem with this way of expressing properties using Prop.forAll: the use of brackets to demarcate the anonymous multi-line function. Notice how the opening bracket appears to the left of the fun keyword, while the closing bracket appears eleven lines down, after the end of the expression Dead =! actual.
While it isn't pretty, I haven't found it that bad for readability, either. When you're writing the property, however, this syntax is cumbersome.
Issues writing the code #
After having written Prop.forAll nc, you start writing (fun neighborCount -> ). After you've typed the closing bracket, you have to move your cursor one place to the left. When entering new lines, you have to keep diligent, making sure that the closing bracket is always to the right of your cursor.
If you ever need to use a nested level of brackets within that anonymous function, your vigilance will be tested to its utmost. When I wrote the above property, for example, when I reached the point where I wanted to call the calculateNextState function, I wrote:
let actual : State =
calculateNextState (
The editor detects that I'm writing an opening bracket, but since I have the closing bracket to the immediate right of the cursor, this is what happens:
let actual : State =
calculateNextState ()
What normally happens when you type an opening bracket is that the editor automatically inserts a closing bracket to the right of your cursor, but in this case, it doesn't do that. There's already a closing bracket immediately to the right of the cursor, and the editor assumes that this bracket belongs to the opening bracket I just typed. What it really should have done was this:
let actual : State = calculateNextState ())
The editor doesn't have a chance, though, because at this point, I'm still typing, and the code doesn't compile. None of the alternatives compile at this point. You can't really blame the editor.
To make a long story short, enclosing a multi-line anonymous function in brackets is a source of errors. If only there was a better alternative.
Backward pipe #
The solution started to dawn on me because I've been learning Haskell for the last half year, and in Haskell, you often use the $ operator to get rid of unwanted brackets. F# has an equivalent operator: <|, the backward pipe operator.
I rarely use the <| operator in F#, but in this case, it works really well:
[<Property(QuietOnSuccess = true)>] let ``Any live cell with > 3 live neighbors dies`` (cell : int * int) = let nc = Gen.elements [4..8] |> Arb.fromGen Prop.forAll nc <| fun neighborCount -> let liveNeighbors = cell |> findNeighbors |> shuffle |> Seq.take neighborCount |> Seq.toList let actual : State = calculateNextState (cell :: liveNeighbors |> shuffle |> set) cell Dead =! actual
Notice that there's no longer any need for a closing bracket after Dead =! actual.
Summary #
When the last argument passed to a function is another function, you can replace the brackets with a single application of the <| operator. I only use this operator sparingly, but in the case of in-line ad-hoc FsCheck Arbitraries, I find it useful.
Addendum 2016-05-17: You can get rid of the nc value with TIE fighter infix notation.
Types + Properties = Software: other properties
Even simple functions may have properties that can be expressed independently of the implementation.
This article is the eighth in a series of articles that demonstrate how to develop software using types and properties. In previous articles, you've seen an extensive example of how to solve the Tennis Kata with type design and Property-Based Testing. Specifically, in the third article, you saw the introduction of a function called other. In that article we didn't cover that function with automatic tests, but this article rectifies that omission.
The source code for this article series is available on GitHub.
Implementation duplication #
The other function is used to find the opponent of a player. The implementation is trivial:
let other = function PlayerOne -> PlayerTwo | PlayerTwo -> PlayerOne
Unless you're developing software where lives, or extraordinary amounts of money, are at stake, you probably need not test such a trivial function. I'm still going to show you how you could do this, mostly because it's a good example of how Property-Based Testing can sometimes help you test the behaviour of a function, instead of the implementation.
Before I show you that, however, I'll show you why example-based unit tests are inadequate for testing this function.
You could test this function using examples, and because the space of possible input is so small, you can even cover it in its entirety:
[<Fact>] let ``other than playerOne returns correct result`` () = PlayerTwo =! other PlayerOne [<Fact>] let ``other than playerTwo returns correct result`` () = PlayerOne =! other PlayerTwo
The =! operator is a custom operator defined by Unquote, an assertion library. You can read the first expression as player two must equal other than player one.
The problem with these tests, however, is that they don't state anything not already stated by the implementation itself. Basically, what these tests state is that if the input is PlayerOne, the output is PlayerTwo, and if the input is PlayerTwo, the output is PlayerOne.
That's exactly what the implementation does.
The only important difference is that the implementation of other states this relationship more succinctly than the tests.
It's as though the tests duplicate the information already in the implementation. How does that add value?
Sometimes, this can be the only way to cover functionality with tests, but in this case, there's an alternative.
Behaviour instead of implementation #
Property-Based Testing inspires you to think about the observable behaviour of a system, rather than the implementation. Which properties do the other function have?
If you call it with a Player value, you'd expect it to return a Player value that's not the input value:
[<Property>] let ``other returns a different player`` player = player <>! other player
The <>! operator is another custom operator defined by Unquote. You can read the expression as player must not equal other player.
This property alone doesn't completely define the behaviour of other, but combined with the next property, it does:
[<Property>] let ``other other returns same player`` player = player =! other (other player)
If you call other, and then you take the output of that call and use it as input to call other again, you should get the original Player value. This property is an excellent example of what Scott Wlaschin calls there and back again.
These two properties, together, describe the behaviour of the other function, without going into details about the implementation.
Summary #
You've seen a simple example of how to describe the properties of a function without resorting to duplicating the implementation in the tests. You may not always be able to do this, but it always feels right when you can.
If you're interested in learning more about Property-Based Testing, you can watch my introduction to Property-based Testing with F# Pluralsight course.
Types + Properties = Software: finite state machine
How to compose the tennis game code into a finite state machine.
This article is the seventh in a series of articles that demonstrate how to develop software using types and properties. In the previous article, you saw how to define the initial state of a tennis game. In this article, you'll see how to define the tennis game as a finite state machine.
The source code for this article series is available on GitHub.
Scoring a sequence of balls #
Previously, you saw how to score a game in an ad-hoc fashion:
> let firstBall = score newGame PlayerTwo;;
val firstBall : Score = Points {PlayerOnePoint = Love;
PlayerTwoPoint = Fifteen;}
> let secondBall = score firstBall PlayerOne;;
val secondBall : Score = Points {PlayerOnePoint = Fifteen;
PlayerTwoPoint = Fifteen;}
You'll quickly get tired of that, so you may think that you can do something like this instead:
> newGame |> (fun s -> score s PlayerOne) |> (fun s -> score s PlayerTwo);;
val it : Score = Points {PlayerOnePoint = Fifteen;
PlayerTwoPoint = Fifteen;}
That does seem a little clumsy, though, but it's not really what you need to be able to do either. What you'd probably appreciate more is to be able to calculate the score given a sequence of wins. A sequence of wins is a list of Player values; for instance, [PlayerOne; PlayerOne; PlayerTwo] means that player one won the first two balls, and then player two won the third ball.
Since we already know the initial state of a game, and how to calculate the score for each ball, it's a one-liner to calculate the score for a sequence of balls:
let scoreSeq wins = Seq.fold score newGame wins
This function has the type Player seq -> Score. It uses newGame as the initial state of a left fold over the wins. The aggregator function is the score function.
You can use it with a sequence of wins, like this:
> scoreSeq [PlayerOne; PlayerOne; PlayerTwo];;
val it : Score = Points {PlayerOnePoint = Thirty;
PlayerTwoPoint = Fifteen;}
Since player one won the first two balls, and player two then won the third ball, the score at that point is thirty-fifteen.
Properties for the state machine #
Can you think of any properties for the scoreSeq function?
There are quite a few, it turns out. It may be a good exercise if you pause reading for a couple of minutes, and try to think of some properties.
If you have a list of properties, you can compare them with the ones I though of.
Before we start, you may find the following helper functions useful:
let isPoints = function Points _ -> true | _ -> false let isForty = function Forty _ -> true | _ -> false let isDeuce = function Deuce -> true | _ -> false let isAdvantage = function Advantage _ -> true | _ -> false let isGame = function Game _ -> true | _ -> false
These can be useful to express some properties, but I don't think that they are generally useful, so I put them together with the test code.
Limited win sequences #
If you look at short win sequences, you can already say something about them. For instance, it takes at least four balls to finish a game, so you know that if you have fewer wins than four, the game must still be on-going:
[<Property>] let ``A game with less then four balls isn't over`` (wins : Player list) = let actual : Score = wins |> Seq.truncate 3 |> scoreSeq test <@ actual |> (not << isGame) @>
The built-in function Seq.truncate returns only the n (in this case: 3) first elements of a sequence. If the sequence is shorter than that, then the entire sequence is returned. The use of Seq.truncate 3 guarantees that the sequence passed to scoreSeq is no longer than three elements long, no matter what FsCheck originally generated.
Likewise, you can't reach deuce in less than six balls:
[<Property>] let ``A game with less than six balls can't be Deuce`` (wins : Player list) = let actual = wins |> Seq.truncate 5 |> scoreSeq test <@ actual |> (not << isDeuce) @>
This is similar to the previous property. There's one more in that family:
[<Property>] let ``A game with less than seven balls can't have any player with advantage`` (wins : Player list) = let actual = wins |> Seq.truncate 6 |> scoreSeq test <@ actual |> (not << isAdvantage) @>
It takes at least seven balls before the score can reach a state where one of the players have the advantage.
Longer win sequences #
Conversely, you can also express some properties related to win sequences of a minimum size. This one is more intricate to express with FsCheck, though. You'll need a sequence of Player values, but the sequence should be guaranteed to have a minimum length. There's no built-in in function for that in FsCheck, so you need to define it yourself:
// int -> Gen<Player list> let genListLongerThan n = let playerGen = Arb.generate<Player> let nPlayers = playerGen |> Gen.listOfLength (n + 1) let morePlayers = playerGen |> Gen.listOf Gen.map2 (@) nPlayers morePlayers
This function takes an exclusive minimum size, and returns an FsCheck generator that will generate Player lists longer than n. It does that by combining two of FsCheck's built-in generators. Gen.listOfLength generates lists of the requested length, and Gen.listOf generates all sorts of lists, including the empty list.
Both nPlayers and morePlayers are values of the type Gen<Player list>. Using Gen.map2, you can concatenate these two list generators using F#'s built-in list concatenation operator @. (All operators in F# are also functions. The (@) function has the type 'a list -> 'a list -> 'a list.)
With the genListLongerThan function, you can now express properties related to longer win sequences. As an example, if the players have played more than four balls, the score can't be a Points case:
[<Property>] let ``A game with more than four balls can't be Points`` () = let moreThanFourBalls = genListLongerThan 4 |> Arb.fromGen Prop.forAll moreThanFourBalls (fun wins -> let actual = scoreSeq wins test <@ actual |> (not << isPoints) @>)
As previously explained, Prop.forAll expresses a property that must hold for all lists generated by moreThanFourBalls.
Correspondingly, if the players have played more than five balls, the score can't be a Forty case:
[<Property>] let ``A game with more than five balls can't be Forty`` () = let moreThanFiveBalls = genListLongerThan 5 |> Arb.fromGen Prop.forAll moreThanFiveBalls (fun wins -> let actual = scoreSeq wins test <@ actual |> (not << isForty) @>)
This property is, as you can see, similar to the previous example.
Shortest completed game #
Do you still have your list of tennis game properties? Let's see if you thought of this one. The shortest possible way to complete a tennis game is when one of the players wins four balls in a row. This property should hold for all (two) players:
[<Property>] let ``A game where one player wins all balls is over in four balls`` (player) = let fourWins = Seq.init 4 (fun _ -> player) let actual = scoreSeq fourWins let expected = Game player expected =! actual
This test first defines fourWins, of the type Player seq. It does that by using the player argument created by FsCheck, and repeating it four times, using Seq.init.
The =! operator is a custom operator defined by Unquote, an assertion library. You can read the expression as expected must equal actual.
Infinite games #
The tennis scoring system turns out to be a rich domain. There are still more properties. We'll look at a final one, because it showcases another way to use FsCheck's API, and then we'll call it a day.
An interesting property of the tennis scoring system is that a game isn't guaranteed to end. It may, theoretically, continue forever, if players alternate winning balls:
[<Property>] let ``A game where players alternate never ends`` firstWinner = let alternateWins = firstWinner |> Gen.constant |> Gen.map (fun p -> [p; other p]) |> Gen.listOf |> Gen.map List.concat |> Arb.fromGen Prop.forAll alternateWins (fun wins -> let actual = scoreSeq wins test <@ actual |> (not << isGame) @>)
This property starts by creating alternateWins, an Arbitrary<Player list>. It creates lists with alternating players, like [PlayerOne; PlayerTwo], or [PlayerTwo; PlayerOne; PlayerTwo; PlayerOne; PlayerTwo; PlayerOne]. It does that by using the firstWinner argument (of the type Player) as a constant starting value. It's only constant within each test, because firstWinner itself varies.
Using that initial Player value, it then proceeds to map that value to a list with two elements: the player, and the other player, using the other function. This creates a Gen<Player list>, but piped into Gen.listOf, it becomes a Gen<Player list list>. An example of the sort of list that would generate could be [[PlayerTwo; PlayerOne]; [PlayerTwo; PlayerOne]]. Obviously, you need to flatten such lists, which you can do with List.concat; you have to do it inside of a Gen, though, so it's Gen.map List.concat. That gives you a Gen<Player list>, which you can finally turn into an Arbitrary<Player list> with Arb.fromGen.
This enables you to use Prop.forAll with alternateWins to express that regardless of the size of such alternating win lists, the game never reaches a Game state.
Summary #
In this article, you saw how to implement a finite state machine for calculating tennis scores. It's a left fold over the score function, always starting in the initial state where both players are at love. You also saw how you can express various properties that hold for a game of tennis.
In this article series, you've seen an extensive example of how to design with types and properties. You may have noticed that out of the six example articles so far, only the first one was about designing with types, and the next five articles were about Property-Based Testing. It looks as though not much is gained from designing with types.
On the contrary, much is gained by designing with types, but you don't see it, exactly because it's efficient. If you don't design with types in order to make illegal states unrepresentable, you'd have to write even more automated tests in order to test what happens when input is invalid. Also, if illegal states are representable, it would have been much harder to write Property-Based Tests. Instead of trusting that FsCheck generates only valid values based exclusively on the types, you'd have to write custom Generators or Arbitraries for each type of input. That would have been even more work than you've seen in this articles series.
Designing with types makes Property-Based Testing a comfortable undertaking. Together, they enable you to develop software that you can trust.
If you're interested in learning more about Property-Based Testing, you can watch my introduction to Property-based Testing with F# Pluralsight course.
Comments
>" This includes protecting the invariants of objects. For Entities, it means that an Entity must always have a stable and enduring identity."
Great points.
However, how would you get from void Create(string firstName, string lastName) to User Read(int id) especially if an entity doesn't have a natural key?
Vladimir, thank you for writing. Exactly how you get the ID depends on the exact requirements of your application. As an example, if you're developing a web application, you may simply be creating the user, and then that's the end of the HTTP request. Meanwhile, asynchronously, a background process is performing the actual database insertion and subsequently sends an email to the user with a link to click in order to verify the email. In that link is the generated ID, so when your server receives the next request, you already know the ID.
That may not always be a practical strategy, but I started to describe this technique, because my experience with CQRS and REST tells me that you have lots of options for communicating state and identity. Still, if you need an ID straight away, you can always pass a GUID as a correlation ID, so that you can later find the Entity you created. That's the technique I originally described.
Thank you for the reply.
BTW, ORMs don't require you to make Ids externally assignable. You can have ORMs assign Ids for you internally, and that's a preferred design in most cases. Also, you can make the Id setters non-public and thus protect entities' encapsulation. There still are issues, of course. Namely, entities don't have established identities until they are saved to the DB or added to the Context/Session. But this issue isn't that big comparing to what you described in the post.
I was skimming through the comments and had this nagging suspicion that the entire topic of, "how do I get my ID with CQS", was a false dichotomy. The premise that the "ID" stored in the database is the "identity" of the object or entity I think can be challenged.
I prefer to look at the "ID" as purely database implementation details. Assuming the repository is SQL, the key type preferred by db admins will be the smallest-reasonable monotonically increasing ID... namely INT. But switch to Amazon DynamoDB and that conclusion flies out the window. With their totally managed and distributed indexes and hashing algorithms... GUID's would be fantastic.
What I WISH developers would do is take the time to think about their Domain and form their own understanding of what truly defines the identity of the entity. Whatever decision they make, it translates to a "natural key" for persistence.
Natural keys are great because a developer has promised, "these combinations of properties uniquely identify entities of type T". The result is unique indexes that can have dramatic performance benefits... outside the application benefit of being able to construct your repository like this:
As for the "ID" field? Who cares?... that's up to the database. The developer already knows exactly how to find this T. I wouldn't even include it on the domain model object.
This would give db admins options. We can look at the natural key size, evaluate how it is queried and order it appropriately. We can decide if the nature of typical joins merit clustering (as is common with Header -> Detail style records). If needed, we can optimize lookup of large keys through stored procedures and computed persisted checksum columns. Nullability and default values become part of design discussions on the front end. Finally, if we feel like it... we can assign a surrogate key, or use a GUID. The developer doesn't care, our dba's do.
Thinking back to my last 6 or so development projects, I've never had to "get the id". The next operation is always a matter of query or command on unique criteria I was responsible for creating in the first place. This approach makes my dba's happy.
Ethan, thank you for your comment; I agree with 90 percent of what you wrote, which is also the reason I prefer the options I previously outlined.
Entities don't always have 'natural IDs', though, and even when they have, experience has taught me not to use them. Is a person's email address a natural ID? It is, but people change email addresses.
Here in Denmark, for many years many public software systems have used the Danish personal identification numbers as natural keys, as they were assumed to be nationally unique and stable. These numbers encode various information about a person, such as birth-date and sex. A few inhabitants, however, get sex change operations, and then desire a new number that corresponds to their new sex. Increasingly, these requests are being granted. New ID, same person.
I've seen such changes happen so many times during my career that I've become distrustful of 'natural IDs'. If I need to identify an Entity, I usually attach a GUID to it.
With regard to entity identification, I'm not sure stability matters. Who cares if the email address changes... as long as it is unique? Uniqueness is sufficient for the application to identify the entity. I agree from a strictly database standpoint; I'm not thrilled about assigning a primary key to a natural key due to the stability issue across relationships. But that does not invalidate the key for use as an identity in my mind. Both indexes can exist, both be unique, and each with their own purpose.
Also, it is difficult for me to envision a scenario where there is no natural key. If there are two rows where the only difference is an auto-incrementing integer, or different guids... what is the meaning of one row versus the next? That meaning cannot, by definition, be contained in the surrogate keys... they add nothing to the business meaning of the entity.
I feel I've derailed the discussion into database modeling theory, and we are talking about CQS with "create the entity" as a stimulating use case. Suffice it to say, IMHO, if (a, b) is unique today according to business rule, than it should be my unique identity from a developer perspective. By leaving row identifiers up to DBA's and dropping them from my domain model entirely, I minimize impact to those technical implementations when the business rules change such that (a, b, c) becomes the new unique, and, preserve CQS in the process.
SQL Server has a feature called 'sequence' (from 2012 and above if I'm not mistaken). We can use sequence to generate unique int ID instead of using identity column. The sequence can be called without inserting row to the table. We can create a service that call this sequence to generate the next running number. The UI code can call this running number generator service to get a new ID, and then send the new entity complete with ID to the create service.