When properties are easier than examples by Mark Seemann
Sometimes, describing the properties of a function is easier than coming up with examples.
Instead of the term test-driven development you may occasionally encounter the phrase example-driven development. The idea is that each test is an example of how the system under test ought to behave. As you add more tests, you add more examples.
I've noticed that beginners often find it difficult to come up with good examples. This is the reason I've developed the Devil's advocate technique. It's meant as a heuristic that may help you identify the next good example. It's particularly effective if you combine it with the Transformation Priority Premise (TPP) and equivalence partitioning.
I've noticed, however, that translating concrete examples into code is not always straightforward. In the following, I'll describe an experience I had in 2020 while developing an online restaurant reservation system.
The code shown here is part of the sample code base that accompanies my book Code That Fits in Your Head.
Problem outline #
I'm going to start by explaining what it was that I was trying to do. I wanted to present the maître d' (or other restaurant staff) with a schedule of a day's reservations. It should take the form of a list of time entries, one entry for every time one or more new reservations would start. I also wanted to list, for each entry, all reservations that were currently ongoing, or would soon start. Here's a simple example, represented as JSON:
"date": "2023-08-23", "entries": [ { "time": "20:00:00", "reservations": [ { "id": "af5feb35f62f475cb02df2a281948829", "at": "2023-08-23T20:00:00.0000000", "email": "crystalmeth@example.net", "name": "Crystal Metheney", "quantity": 3 }, { "id": "eae39bc5b3a7408eb2049373b2661e32", "at": "2023-08-23T20:30:00.0000000", "email": "x.benedict@example.org", "name": "Benedict Xavier", "quantity": 4 } ] }, { "time": "20:30:00", "reservations": [ { "id": "af5feb35f62f475cb02df2a281948829", "at": "2023-08-23T20:00:00.0000000", "email": "crystalmeth@example.net", "name": "Crystal Metheney", "quantity": 3 }, { "id": "eae39bc5b3a7408eb2049373b2661e32", "at": "2023-08-23T20:30:00.0000000", "email": "x.benedict@example.org", "name": "Benedict Xavier", "quantity": 4 } ] } ]
To keep the example simple, there are only two reservations for that particular day: one for 20:00 and one for 20:30. Since something happens at both of these times, both time has an entry. My intent isn't necessarily that a user interface should show the data in this way, but I wanted to make the relevant data available so that a user interface could show it if it needed to.
The first entry for 20:00 shows both reservations. It shows the reservation for 20:00 for obvious reasons, and it shows the reservation for 20:30 to indicate that the staff can expect a party of four at 20:30. Since this restaurant runs with a single seating per evening, this effectively means that although the reservation hasn't started yet, it still reserves a table. This gives a user interface an opportunity to show the state of the restaurant at that time. The table for the 20:30 party isn't active yet, but it's effectively reserved.
For restaurants with shorter seating durations, the schedule should reflect that. If the seating duration is, say, two hours, and someone has a reservation for 20:00, you can sell that table to another party at 18:00, but not at 18:30. I wanted the functionality to take such things into account.
The other entry in the above example is for 20:30. Again, both reservations are shown because one is ongoing (and takes up a table) and the other is just starting.
Desired API #
A major benefit of test-driven development (TDD) is that you get fast feedback on the API you intent for the system under test (SUT). You write a test against the intended API, and besides a pass-or-fail result, you also learn something about the interaction between client code and the SUT. You often learn that the original design you had in mind isn't going to work well once it meets the harsh realities of an actual programming language.
In TDD, you often have to revise the design multiple times during the process.
This doesn't mean that you can't have a plan. You can't write the initial test if you have no inkling of what the API should look like. For the schedule feature, I did have a plan. It turned out to hold, more or less. I wanted the API to be a method on a class called MaitreD, which already had these four fields and the constructors to support them:
public TimeOfDay OpensAt { get; } public TimeOfDay LastSeating { get; } public TimeSpan SeatingDuration { get; } public IEnumerable<Table> Tables { get; }
I planned to implement the new feature as a new instance method on that class:
public IEnumerable<Occurrence<IEnumerable<Table>>> Schedule(IEnumerable<Reservation> reservations)
This plan turned out to hold in general, although I ultimately decided to simplify the return type by getting rid of the Occurrence container. It's going to be present throughout this article, however, so I need to briefly introduce it. I meant to use it as a generic container of anything, but with an time-stamp associated with the value:
public sealed class Occurrence<T> { public Occurrence(DateTime at, T value) { At = at; Value = value; } public DateTime At { get; } public T Value { get; } public Occurrence<TResult> Select<TResult>(Func<T, TResult> selector) { if (selector is null) throw new ArgumentNullException(nameof(selector)); return new Occurrence<TResult>(At, selector(Value)); } public override bool Equals(object? obj) { return obj is Occurrence<T> occurrence && At == occurrence.At && EqualityComparer<T>.Default.Equals(Value, occurrence.Value); } public override int GetHashCode() { return HashCode.Combine(At, Value); } }
You may notice that due to the presence of the Select method this is a functor.
There's also a little extension method that we may later encounter:
public static Occurrence<T> At<T>(this T value, DateTime at) { return new Occurrence<T>(at, value); }
The plan, then, is to return a collection of occurrences, each of which may contain a collection of tables that are relevant to include at that time entry.
Examples #
When I embarked on developing this feature, I thought that it was a good fit for example-driven development. Since the input for Schedule requires a collection of Reservation objects, each of which comes with some data, I expected the test cases to become verbose. So I decided to bite the bullet right away and define test cases using xUnit.net's [ClassData] feature. I wrote this test:
[Theory, ClassData(typeof(ScheduleTestCases))] public void Schedule( MaitreD sut, IEnumerable<Reservation> reservations, IEnumerable<Occurrence<Table[]>> expected) { var actual = sut.Schedule(reservations); Assert.Equal( expected.Select(o => o.Select(ts => ts.AsEnumerable())), actual); }
This is almost as simple as it can be: Call the method and verify that expected is equal to actual. The only slightly complicated piece is the nested projection of expected from IEnumerable<Occurrence<Table[]>> to IEnumerable<Occurrence<IEnumerable<Table>>>. There are ugly reasons for this that I don't want to discuss here, since they have no bearing on the actual topic, which is coming up with tests.
I also added the ScheduleTestCases class and a single test case:
private class ScheduleTestCases : TheoryData<MaitreD, IEnumerable<Reservation>, IEnumerable<Occurrence<Table[]>>> { public ScheduleTestCases() { // No reservations, so no occurrences: Add(new MaitreD( TimeSpan.FromHours(18), TimeSpan.FromHours(21), TimeSpan.FromHours(6), Table.Communal(12)), Array.Empty<Reservation>(), Array.Empty<Occurrence<Table[]>>()); } }
The simplest implementation that passed that test was this:
public IEnumerable<Occurrence<IEnumerable<Table>>> Schedule(IEnumerable<Reservation> reservations) { yield break; }
Okay, hardly rocket science, but this was just a test case to get started. So I added another one:
private void SingleReservationCommunalTable() { var table = Table.Communal(12); var r = Some.Reservation; Add(new MaitreD( TimeSpan.FromHours(18), TimeSpan.FromHours(21), TimeSpan.FromHours(6), table), new[] { r }, new[] { new[] { table.Reserve(r) }.At(r.At) }); }
This test case adds a single reservation to a restaurant with a single communal table. The expected result is now a single occurrence with that reservation. In true TDD fashion, this new test case caused a test failure, and I now had to adjust the Schedule method to pass all tests:
public IEnumerable<Occurrence<IEnumerable<Table>>> Schedule(IEnumerable<Reservation> reservations) { if (reservations.Any()) { var r = reservations.First(); yield return new[] { Table.Communal(12).Reserve(r) }.AsEnumerable().At(r.At); } yield break; }
You might have wanted to jump to something prettier right away, but I wanted to proceed according to the Devil's advocate technique. I was concerned that I was going to mess up the implementation if I moved too fast.
And that was when I basically hit a wall.
Property-based testing to the rescue #
I couldn't figure out how to proceed from there. Which test case ought to be the next? I wanted to follow the spirit of the TPP and pick a test case that would cause another incremental step in the right direction. The sheer number of possible combinations overwhelmed me, though. Should I adjust the reservations? The table configuration for the MaitreD class? The SeatingDuration?
It's possible that you'd be able to conjure up the perfect next test case, but I couldn't. I actually let it stew for a couple of days before I decided to give up on the example-driven approach. While I couldn't see a clear path forward with concrete examples, I had a vivid vision of how to proceed with property-based testing.
I left the above tests in place and instead added a new test class to my code base. Its only purpose: to test the Schedule method. The test method itself is only a composition of various data definitions and the actual test code:
[Property] public Property Schedule() { return Prop.ForAll( GenReservation.ArrayOf().ToArbitrary(), ScheduleImp); }
This uses FsCheck 2.14.3, which is written in F# and composes better if you also write the tests in F#. In order to make things a little more palatable for C# developers, I decided to implement the building blocks for the property using methods and class properties.
The ScheduleImp method, for example, actually implements the test. This method runs a hundred times (FsCheck's default value) with randomly generated input values:
private static void ScheduleImp(Reservation[] reservations) { // Create a table for each reservation, to ensure that all // reservations can be allotted a table. var tables = reservations.Select(r => Table.Standard(r.Quantity)); var sut = new MaitreD( TimeSpan.FromHours(18), TimeSpan.FromHours(21), TimeSpan.FromHours(6), tables); var actual = sut.Schedule(reservations); Assert.Equal( reservations.Select(r => r.At).Distinct().Count(), actual.Count()); }
The step you see in the first line of code is an example of a trick that I find myself doing often with property-based testing: instead of trying to find some good test values for a particular set of circumstances, I create a set of circumstances that fits the randomly generated test values. As the code comment explains, given a set of Reservation values, it creates a table that fits each reservation. In that way I ensure that all the reservations can be allocated a table.
I'll soon return to how those random Reservation values are generated, but first let's discuss the rest of the test body. Given a valid MaitreD object it calls the Schedule method. In the assertion phase, it so far only verifies that there's as many time entries in actual as there are distinct At values in reservations.
That's hardly a comprehensive description of the SUT, but it's a start. The following implementation passes both the new property, as well as the two examples above.
public IEnumerable<Occurrence<IEnumerable<Table>>> Schedule(IEnumerable<Reservation> reservations) { return from r in reservations group Table.Communal(12).Reserve(r) by r.At into g select g.AsEnumerable().At(g.Key); }
I know that many C# programmers don't like query syntax, but I've always had a soft spot for it. I liked it, but wasn't sure that I'd be able to keep it up as I added more constraints to the property.
Generators #
Before we get to that, though, I promised to show you how the random reservations are generated. FsCheck has an API for that, and it's also query-syntax-friendly:
private static Gen<Email> GenEmail => from s in Arb.Default.NonWhiteSpaceString().Generator select new Email(s.Item); private static Gen<Name> GenName => from s in Arb.Default.StringWithoutNullChars().Generator select new Name(s.Item); private static Gen<Reservation> GenReservation => from id in Arb.Default.Guid().Generator from d in Arb.Default.DateTime().Generator from e in GenEmail from n in GenName from q in Arb.Default.PositiveInt().Generator select new Reservation(id, d, e, n, q.Item);
GenReservation is a generator of Reservation values (for a simplified explanation of how such a generator might work, see The Test Data Generator functor). It's composed from smaller generators, among these GenEmail and GenName. The rest of the generators are general-purpose generators defined by FsCheck.
If you refer back to the Schedule property above, you'll see that it uses GenReservation to produce an array generator. This is another general-purpose combinator provided by FsCheck. It turns any single-object generator into a generator of arrays containing such objects. Some of these arrays will be empty, which is often desirable, because it means that you'll automatically get coverage of that edge case.
Iterative development #
As I already discovered in 2015 some problems are just much better suited for property-based development than example-driven development. As I expected, this one turned out to be just such a problem. (Recently, Hillel Wayne identified a set of problems with no clear properties as rho problems. I wonder if we should pick another Greek letter for this type of problems that almost ooze properties. Sigma problems? Maybe we should just call them describable problems...)
For the next step, I didn't have to write a completely new property. I only had to add a new assertion, and thereby strengthening the postconditions of the Schedule method:
Assert.Equal( actual.Select(o => o.At).OrderBy(d => d), actual.Select(o => o.At));
I added the above assertion to ScheduleImp after the previous assertion. It simply states that actual should be sorted in ascending order.
To pass this new requirement I added an ordering clause to the implementation:
public IEnumerable<Occurrence<IEnumerable<Table>>> Schedule(IEnumerable<Reservation> reservations) { return from r in reservations group Table.Communal(12).Reserve(r) by r.At into g orderby g.Key select g.AsEnumerable().At(g.Key); }
It passes all tests. Commit to Git. Next.
Table configuration #
If you consider the current implementation, there's much not to like. The worst offence, I think, is that it conjures a hard-coded communal table out of thin air. The method ought to use the table configuration passed to the MaitreD object. This seems like an obvious flaw to address. I therefore added this to the property:
Assert.All(actual, o => AssertTables(tables, o.Value)); } private static void AssertTables( IEnumerable<Table> expected, IEnumerable<Table> actual) { Assert.Equal(expected.Count(), actual.Count()); }
It's just another assertion that uses the helper assertion also shown. As a first pass, it's not enough to cheat the Devil, but it sets me up for my next move. The plan is to assert that no tables are generated out of thin air. Currently, AssertTables only verifies that the actual count of tables in each occurrence matches the expected count.
The Devil easily foils that plan by generating a table for each reservation:
public IEnumerable<Occurrence<IEnumerable<Table>>> Schedule(IEnumerable<Reservation> reservations) { var tables = reservations.Select(r => Table.Communal(12).Reserve(r)); return from r in reservations group r by r.At into g orderby g.Key select tables.At(g.Key); }
This (unfortunately) passes all tests, so commit to Git and move on.
The next move I made was to add an assertion to AssertTables:
Assert.Equal( expected.Sum(t => t.Capacity), actual.Sum(t => t.Capacity));
This new requirement states that the total capacity of the actual tables should be equal to the total capacity of the allocated tables. It doesn't prevent the Devil from generating tables out of thin air, but it makes it harder. At least, it makes it so hard that I found it more reasonable to use the supplied table configuration:
public IEnumerable<Occurrence<IEnumerable<Table>>> Schedule(IEnumerable<Reservation> reservations) { var tables = reservations.Zip(Tables, (r, t) => t.Reserve(r)); return from r in reservations group r by r.At into g orderby g.Key select tables.At(g.Key); }
The implementation of Schedule still cheats because it 'knows' that no tests (except for the degenerate test where there are no reservations) have surplus tables in the configuration. It takes advantage of that knowledge to zip the two collections, which is really not appropriate.
Still, it seems that things are moving in the right direction.
Generated SUT #
Until now, ScheduleImp has been using a hard-coded sut. It's time to change that.
To keep my steps as small as possible, I decided to start with the SeatingDuration since it was currently not being used by the implementation. This meant that I could start randomising it without affecting the SUT. Since this was a code change of middling complexity in the test code, I found it most prudent to move in such a way that I didn't have to change the SUT as well.
I completely extracted the initialisation of the sut to a method argument of the ScheduleImp method, and adjusted it accordingly:
private static void ScheduleImp(MaitreD sut, Reservation[] reservations) { var actual = sut.Schedule(reservations); Assert.Equal( reservations.Select(r => r.At).Distinct().Count(), actual.Count()); Assert.Equal( actual.Select(o => o.At).OrderBy(d => d), actual.Select(o => o.At)); Assert.All(actual, o => AssertTables(sut.Tables, o.Value)); }
This meant that I also had to adjust the calling property:
public Property Schedule() { return Prop.ForAll( GenReservation .ArrayOf() .SelectMany(rs => GenMaitreD(rs).Select(m => (m, rs))) .ToArbitrary(), t => ScheduleImp(t.m, t.rs)); }
You've already seen GenReservation, but GenMaitreD is new:
private static Gen<MaitreD> GenMaitreD(IEnumerable<Reservation> reservations) { // Create a table for each reservation, to ensure that all // reservations can be allotted a table. var tables = reservations.Select(r => Table.Standard(r.Quantity)); return from seatingDuration in Gen.Choose(1, 6) select new MaitreD( TimeSpan.FromHours(18), TimeSpan.FromHours(21), TimeSpan.FromHours(seatingDuration), tables); }
The only difference from before is that the new MaitreD object is now initialised from within a generator expression. The duration is randomly picked from the range of one to six hours (those numbers are my arbitrary choices).
Notice that it's possible to base one generator on values randomly generated by another generator. Here, reservations are randomly produced by GenReservation and merged to a tuple with SelectMany, as you can see above.
This in itself didn't impact the SUT, but set up the code for my next move, which was to generate more tables than reservations, so that there'd be some free tables left after the schedule allocation. I first added a more complex table generator:
/// <summary> /// Generate a table configuration that can at minimum accomodate all /// reservations. /// </summary> /// <param name="reservations">The reservations to accommodate</param> /// <returns>A generator of valid table configurations.</returns> private static Gen<IEnumerable<Table>> GenTables(IEnumerable<Reservation> reservations) { // Create a table for each reservation, to ensure that all // reservations can be allotted a table. var tables = reservations.Select(r => Table.Standard(r.Quantity)); return from moreTables in Gen.Choose(1, 12).Select(Table.Standard).ArrayOf() from allTables in Gen.Shuffle(tables.Concat(moreTables)) select allTables.AsEnumerable(); }
This function first creates standard tables that exactly accommodate each reservation. It then generates an array of moreTables, each fitting between one and twelve people. It then mixes those tables together with the ones that fit a reservation and returns the sequence. Since moreTables can be empty, it's possible that the entire sequence of tables only just accommodates the reservations.
I then modified GenMaitreD to use GenTables:
private static Gen<MaitreD> GenMaitreD(IEnumerable<Reservation> reservations) { return from seatingDuration in Gen.Choose(1, 6) from tables in GenTables(reservations) select new MaitreD( TimeSpan.FromHours(18), TimeSpan.FromHours(21), TimeSpan.FromHours(seatingDuration), tables); }
This provoked a change in the SUT:
public IEnumerable<Occurrence<IEnumerable<Table>>> Schedule(IEnumerable<Reservation> reservations) { return from r in reservations group r by r.At into g orderby g.Key select Allocate(g).At(g.Key); }
The Schedule method now calls a private helper method called Allocate. This method already existed, since it supports the algorithm used to decide whether or not to accept a reservation request.
Rinse and repeat #
I hope that a pattern starts to emerge. I kept adding more and more randomisation to the data generators, while I also added more and more assertions to the property. Here's what it looked like after a few more iterations:
private static void ScheduleImp(MaitreD sut, Reservation[] reservations) { var actual = sut.Schedule(reservations); Assert.Equal( reservations.Select(r => r.At).Distinct().Count(), actual.Count()); Assert.Equal( actual.Select(o => o.At).OrderBy(d => d), actual.Select(o => o.At)); Assert.All(actual, o => AssertTables(sut.Tables, o.Value)); Assert.All( actual, o => AssertRelevance(reservations, sut.SeatingDuration, o)); }
While AssertTables didn't change further, I added another helper assertion called AssertRelevance. I'm not going to show it here, but it checks that each occurrence only contains reservations that overlaps that point in time, give or take the SeatingDuration.
I also made the reservation generator more sophisticated. If you consider the one defined above, one flaw is that it generates reservations at random dates. The chance that it'll generate two reservations that are actually adjacent in time is minimal. To counter this problem, I added a function that would return a generator of adjacent reservations:
/// <summary> /// Generate an adjacant reservation with a 25% chance. /// </summary> /// <param name="reservation">The candidate reservation</param> /// <returns> /// A generator of an array of reservations. The generated array is /// either a singleton or a pair. In 75% of the cases, the input /// <paramref name="reservation" /> is returned as a singleton array. /// In 25% of the cases, the array contains two reservations: the input /// reservation as well as another reservation adjacent to it. /// </returns> private static Gen<Reservation[]> GenAdjacentReservations(Reservation reservation) { return from adjacent in GenReservationAdjacentTo(reservation) from useAdjacent in Gen.Frequency( new WeightAndValue<Gen<bool>>(3, Gen.Constant(false)), new WeightAndValue<Gen<bool>>(1, Gen.Constant(true))) let rs = useAdjacent ? new[] { reservation, adjacent } : new[] { reservation } select rs; } private static Gen<Reservation> GenReservationAdjacentTo(Reservation reservation) { return from minutes in Gen.Choose(-6 * 4, 6 * 4) // 4: quarters/h from r in GenReservation select r.WithDate( reservation.At + TimeSpan.FromMinutes(minutes)); }
Now that I look at it again, I wonder whether I could have expressed this in a simpler way... It gets the job done, though.
I then defined a generator that would either create entirely random reservations, or some with some adjacent ones mixed in:
private static Gen<Reservation[]> GenReservations { get { var normalArrayGen = GenReservation.ArrayOf(); var adjacentReservationsGen = GenReservation.ArrayOf() .SelectMany(rs => Gen .Sequence(rs.Select(GenAdjacentReservations)) .SelectMany(rss => Gen.Shuffle( rss.SelectMany(rs => rs)))); return Gen.OneOf(normalArrayGen, adjacentReservationsGen); } }
I changed the property to use this generator instead:
[Property] public Property Schedule() { return Prop.ForAll( GenReservations .SelectMany(rs => GenMaitreD(rs).Select(m => (m, rs))) .ToArbitrary(), t => ScheduleImp(t.m, t.rs)); }
I could have kept at it longer, but this turned out to be good enough to bring about the change in the SUT that I was looking for.
Implementation #
These incremental changes iteratively brought me closer and closer to an implementation that I think has the correct behaviour:
public IEnumerable<Occurrence<IEnumerable<Table>>> Schedule(IEnumerable<Reservation> reservations) { return from r in reservations group r by r.At into g orderby g.Key let seating = new Seating(SeatingDuration, g.Key) let overlapping = reservations.Where(seating.Overlaps) select Allocate(overlapping).At(g.Key); }
Contrary to my initial expectations, I managed to keep the implementation to a single query expression all the way through.
Conclusion #
This was a problem that I was stuck on for a couple of days. I could describe the properties I wanted the function to have, but I had a hard time coming up with a good set of examples for unit tests.
You may think that using property-based testing looks even more complicated, and I admit that it's far from trivial. The problem itself, however, isn't easy, and while the property-based approach may look daunting, it turned an intractable problem into a manageable one. That's a win in my book.
It's also worth noting that this would all have looked more elegant in F#. There's an object-oriented tax to be paid when using FsCheck from C#.
