An example of state-based testing in Haskell by Mark Seemann
How do you do state-based testing when state is immutable? You use the State monad.
This article is an instalment in an article series about how to move from interaction-based testing to state-based testing. In the previous article, you saw an example of an interaction-based unit test written in C#. The problem that this article series attempts to address is that interaction-based testing can lead to what xUnit Test Patterns calls Fragile Tests, because the tests get coupled to implementation details, instead of overall behaviour.
My experience is that functional programming is better aligned with unit testing because functional design is intrinsically testable. While I believe that functional programming is no panacea, it still seems to me that we can learn many valuable lessons about programming from it.
People often ask me about F# programming: How do I know that my F# code is functional?
I sometimes wonder that myself, about my own F# code. One can certainly choose to ignore such a question as irrelevant, and I sometimes do, as well. Still, in my experience, asking such questions can create learning opportunities.
The best answer that I've found is: Port the F# code to Haskell.
Haskell enforces referential transparency via its compiler. If Haskell code compiles, it's functional. In this article, then, I take the problem from the previous article and port it to Haskell.
The code shown in this article is available on GitHub.
A function to connect two users #
In the previous article, you saw implementation and test coverage of a piece of software functionality to connect two users with each other. This was a simplification of the example running through my two Clean Coders videos, Church Visitor and Preserved in translation.
In contrast to the previous article, we'll start with the implementation of the System Under Test (SUT).
post :: Monad m => (a -> m (Either UserLookupError User)) -> (User -> m ()) -> a -> a -> m (HttpResponse User) post lookupUser updateUser userId otherUserId = do userRes <- first (\case InvalidId -> "Invalid user ID." NotFound -> "User not found.") <$> lookupUser userId otherUserRes <- first (\case InvalidId -> "Invalid ID for other user." NotFound -> "Other user not found.") <$> lookupUser otherUserId connect <- runExceptT $ do user <- ExceptT $ return userRes otherUser <- ExceptT $ return otherUserRes lift $ updateUser $ addConnection user otherUser return otherUser return $ either BadRequest OK connect
This is as direct a translation of the C# code as makes sense. If I'd only been implementing the desired functionality in Haskell, without having to port existing code, I'd designed the code differently.
This post function uses partial application as an analogy to dependency injection, but in order to enable potentially impure operations to take place, everything must happen inside of some monad. While the production code must ultimately run in the IO monad in order to interact with a database, tests can choose to run in another monad.
In the C# example, two dependencies are injected into the class that defines the Post method. In the above Haskell function, these two dependencies are instead passed as function arguments. Notice that both functions return values in the monad m.
The intent of the lookupUser argument is that it'll query a database with a user ID. It'll return the user if present, but it could also return a UserLookupError, which is a simple sum type:
data UserLookupError = InvalidId | NotFound deriving (Show, Eq)
If both users are found, the function connects the users and calls the updateUser function argument. The intent of this 'dependency' is that it updates the database. This is recognisably a Command, since its return type is m () - unit (()) is equivalent to void.
State-based testing #
How do you unit test such a function? How do you use Mocks and Stubs in Haskell? You don't; you don't have to. While the post method can be impure (when m is IO), it doesn't have to be. Functional design is intrinsically testable, but that proposition depends on purity. Thus, it's worth figuring out how to keep the post function pure in the context of unit testing.
While IO implies impurity, most common monads are pure. Which one should you choose? You could attempt to entirely 'erase' the monadic quality of the post function with the Identity monad, but if you do that, you can't verify whether or not updateUser was invoked.
While you could write an ad-hoc Mock using, for example, the Writer monad, it might be a better choice to investigate if something closer to state-based testing would be possible.
In an object-oriented context, state-based testing implies that you exercise the SUT, which mutates some state, and then you verify that the (mutated) state matches your expectations. You can't do that when you test a pure function, but you can examine the state of the function's return value. The State monad is an obvious choice, then.
A Fake database #
Haskell's State monad is parametrised on the state type as well as the normal 'value type', so in order to be able to test the post function, you'll have to figure out what type of state to use. The interactions implied by the post function's lookupUser and updateUser arguments are those of database interactions. A Fake database seems an obvious choice.
For the purposes of testing the post function, an in-memory database implemented using a Map is appropriate:
type DB = Map Integer User
This is simply a dictionary keyed by Integer values and containing User values. You can implement compatible lookupUser and updateUser functions with State DB as the Monad. The updateUser function is the easiest one to implement:
updateUser :: User -> State DB () updateUser user = modify $ Map.insert (userId user) user
This simply inserts the user into the database, using the userId as the key. The type of the function is compatible with the general requirement of User -> m (), since here, m is State DB.
The lookupUser Fake implementation is a bit more involved:
lookupUser :: String -> State DB (Either UserLookupError User) lookupUser s = do let maybeInt = readMaybe s :: Maybe Integer let eitherInt = maybe (Left InvalidId) Right maybeInt db <- get return $ eitherInt >>= maybe (Left NotFound) Right . flip Map.lookup db
First, consider the type. The function takes a String value as an argument and returns a State DB (Either UserLookupError User). The requirement is a function compatible with the type a -> m (Either UserLookupError User). This works when a is String and m is, again, State DB.
The entire function is written in do notation, where the inferred Monad is, indeed, State DB. The first line attempts to parse the String into an Integer. Since the built-in readMaybe function returns a Maybe Integer, the next line uses the maybe function to handle the two possible cases, converting the Nothing case into the Left InvalidId value, and the Just case into a Right value.
It then uses the State module's get function to access the database db, and finally attempt a lookup against that Map. Again, maybe is used to convert the Maybe value returned by Map.lookup into an Either value.
Happy path test case #
This is all you need in terms of Test Doubles. You now have test-specific lookupUser and updateUser functions that you can pass to the post function.
Like in the previous article, you can start by exercising the happy path where a user successfully connects with another user:
testProperty "Users successfully connect" $ \ user otherUser -> runStateTest $ do put $ Map.fromList [toDBEntry user, toDBEntry otherUser] actual <- post lookupUser updateUser (show $ userId user) (show $ userId otherUser) db <- get return $ isOK actual && any (elem otherUser . connectedUsers) (Map.lookup (userId user) db)
Here I'm inlining test cases as anonymous functions - this time expressing the tests as QuickCheck properties. I'll later return to the runStateTest helper function, but first I want to focus on the test body itself. It's written in do notation, and specifically, it runs in the State DB monad.
user and otherUser are input arguments to the property. These are both User values, since the test also defines Arbitrary instances for that type (not shown in this article; see the source code repository for details).
The first step in the test is to 'save' both users in the Fake database. This is easily done by converting each User value to a database entry:
toDBEntry :: User -> (Integer, User) toDBEntry = userId &&& id
Recall that the Fake database is nothing but an alias over Map Integer User, so the only operation required to turn a User into a database entry is to extract the key.
The next step in the test is to exercise the SUT, passing the test-specific lookupUser and updateUser Test Doubles to the post function, together with the user IDs converted to String values.
In the assert phase of the test, it first extracts the current state of the database, using the State library's built-in get function. It then verifies that actual represents a 200 OK value, and that the user entry in the database now contains otherUser as a connected user.
Missing user test case #
While there's one happy-path test case, there's four other test cases left. One of these is when the first user doesn't exist:
testProperty "Users don't connect when user doesn't exist" $ \ (Positive i) otherUser -> runStateTest $ do let db = Map.fromList [toDBEntry otherUser] put db let uniqueUserId = show $ userId otherUser + i actual <- post lookupUser updateUser uniqueUserId (show $ userId otherUser) assertPostFailure db actual
What ought to trigger this test case is that the 'first' user doesn't exist, even if the otherUser does exist. For this reason, the test inserts the otherUser into the Fake database.
Since the test is a QuickCheck property, i could be any positive Integer value - including the userId of otherUser. In order to properly exercise the test case, however, you'll need to call the post function with a uniqueUserId - thas it: an ID which is guaranteed to not be equal to the userId of otherUser. There's several options for achieving this guarantee (including, as you'll see soon, the ==> operator), but a simple way is to add a non-zero number to the number you need to avoid.
You then exercise the post function and, as a verification, call a reusable assertPostFailure function:
assertPostFailure :: (Eq s, Monad m) => s -> HttpResponse a -> StateT s m Bool assertPostFailure stateBefore resp = do stateAfter <- get let stateDidNotChange = stateBefore == stateAfter return $ stateDidNotChange && isBadRequest resp
This function verifies that the state of the database didn't change, and that the response value represents a 400 Bad Request HTTP response. This verification doesn't actually verify that the error message associated with the BadRequest case is the expected message, like in the previous article. This would, however, involve a fairly trivial change to the code.
Missing other user test case #
Similar to the above test case, users will also fail to connect if the 'other user' doesn't exist. The property is almost identical:
testProperty "Users don't connect when other user doesn't exist" $ \ (Positive i) user -> runStateTest $ do let db = Map.fromList [toDBEntry user] put db let uniqueOtherUserId = show $ userId user + i actual <- post lookupUser updateUser (show $ userId user) uniqueOtherUserId assertPostFailure db actual
Since this test body is so similar to the previous test, I'm not going to give you a detailed walkthrough. I did, however, promise to describe the runStateTest helper function:
runStateTest :: State (Map k a) b -> b runStateTest = flip evalState Map.empty
Since this is a one-liner, you could also write all the tests by simply in-lining that little expression, but I thought that it made the tests more readable to give this function an explicit name.
It takes any State (Map k a) b and runs it with an empty map. Thus, all State-valued functions, like the tests, must explicitly put data into the state. This is also what the tests do.
Notice that all the tests return State values. For example, the assertPostFailure function returns StateT s m Bool, of which State s Bool is an alias. This fits State (Map k a) b when s is Map k a, which again is aliased to DB. Reducing all of this, the tests are simply functions that return Bool.
Invalid user ID test cases #
Finally, you can also cover the two test cases where one of the user IDs is invalid:
testProperty "Users don't connect when user Id is invalid" $ \ s otherUser -> isIdInvalid s ==> runStateTest $ do let db = Map.fromList [toDBEntry otherUser] put db actual <- post lookupUser updateUser s (show $ userId otherUser) assertPostFailure db actual , testProperty "Users don't connect when other user Id is invalid" $ \ s user -> isIdInvalid s ==> runStateTest $ do let db = Map.fromList [toDBEntry user] put db actual <- post lookupUser updateUser (show $ userId user) s assertPostFailure db actual
Both of these properties take a String value s as input. When QuickCheck generates a String, that could be any String value. Both tests require that the value is an invalid user ID. Specifically, it mustn't be possible to parse the string into an Integer. If you don't constrain QuickCheck, it'll generate various strings, including e.g. "8" and other strings that can be parsed as numbers.
In the above "Users don't connect when user doesn't exist" test, you saw how one way to explicitly model constraints on data is to project a seed value in such a way that the constraint always holds. Another way is to use QuickCheck's built-in ==> operator to filter out undesired values. In this example, both tests employ the isIdInvalid function:
isIdInvalid :: String -> Bool isIdInvalid s = let userInt = readMaybe s :: Maybe Integer in isNothing userInt
Using isIdInvalid with the ==> operator guarantees that s is an invalid ID.
Summary #
While state-based testing may, at first, sound incompatible with strictly functional programming, it's not only possible with the State monad, but even, with good language support, easily done.
The tests shown in this article aren't concerned with the interactions between the SUT and its dependencies. Instead, they compare the initial state with the state after exercising the SUT. Comparing values, even complex data structures such as maps, tends to be trivial in functional programming. Immutable values typically have built-in structural equality (in Haskell signified by the automatic Eq type class), which makes comparing them trivial.
Now that we know that state-based testing is possible even with Haskell's enforced purity, it should be clear that we can repeat the feat in F#.
