Creating Booleans With AutoFixture

Sunday, 19 April 2009 17:40:36 UTC

(Just back after 14 days in Morocco, I'll pick up where I left…)

The last group of built-in special creation algorithms for AutoFixture, besides strings and numbers, concerns Booleans.

The default algorithm is to alternate between true and false, starting with true; i.e. the first time you invoke

fixture.CreateAnonymous<bool>();

it will return true, the next time false, then true again, etc.

The reason I chose this algorithm was because I wanted to ensure that the first time AutoFixture creates an anonymous Boolean, it will return true, which is different than the default (false, in case you were in doubt). This gives us better assurance that a given constructor argument or property is being assigned a real value instead of the default.

Like with numbers, using the overload that takes a seed has no effect, and the seed is simply ignored.

fixture.CreateAnonymous(true);

In other words, the above method is still going to return false every second time, so it doesn't really make sense to use this overload at all for Booleans (or numbers).


Creating Numbers With AutoFixture

Friday, 03 April 2009 21:07:13 UTC

Previously, we saw how AutoFixture creates strings. In this post, I'll explain how it creates numbers. Once again, the algorithm that I'll explain here is the default algorithm, but if you don't like it, you can replace it with something else.

It's very simple: Numbers are returned in the ordered sequence of natural numbers (1, 2, 3, 4, 5, …). The first time you call

fixture.CreateAnonymous<int>();

the returned number will be 1, the second time 2, etc.

The reason I chose that particular algorithm is because it creates numbers that we, as humans, find… well… natural!

A lot of the domains we model work with natural numbers, and even if you write an API where negative numbers are allowed, it's fairly unlikely that positive numbers will not be allowed. Thus, in most cases, small positive integers tend to be ‘nice' values in most APIs - and recall that when we do TDD, we focus on the Happy Path, so it's important to pick values that take us down that path.

Using the overload that takes a seed, like this:

fixture.CreateAnonymous(42);

has no effect - the seed (in this case 42) is simply ignored, so if you call this after first calling the parameterless overload twice, the return number is going to be 3.

Each number type, however, has its own sequence, so even if you've been creating a few Int32 instances like above,

fixture.CreateAnonymous<decimal>();

will return 1.

The following number types all work that way:

  • Byte
  • Decimal
  • Double
  • Int16
  • Int32
  • Int64
  • SByte
  • Single
  • UInt16
  • UInt32
  • UInt64

Comments

florin
Having them sequence ordered sounds like there will not be so "non-deterministic". Wouldn't a random number be more appropriate as it would allow testing of different cases each time. Sure it would not be reproducible but at least you will get to know that there might be something wrong with the code under test.
2010-05-21 10:29 UTC
The idea about constrained non-determinism is that although test values are essentially unknown, two consecutive test runs must traverse the same branch of source code each time. In some SUTs different integer values may cause different branches to be exectured for each test run, so I found it safer to use a deterministic sequence.

For complex objects (say: a class with two writable properties of the same number type) the ordering of assignment is undefined even though the sequence of numbers is deterministic. In other words, if you have the properties Number1 and Number2, you know one gets the value 1 and the other gets the value 2, but you don't know which one gets what (it's probably deterministic anyway because Reflection is likely to always return properties in the same order, but AutoFixture does nothing to explicitly order properties and fields).

Another similar strategy could be to use random numbers from a constant seed. This would give you another stable sequence of numbers.

If none of the above is a concern (numbers do not influence the execution path) you can also choose to use pure random numbers.

In any case you can use the Register method to override the default behavior for a given type, so it's entirely possible to set it up with random numbers instead of a rising sequence.
2010-05-21 12:33 UTC
If you offer an overload for the constructor that takes a seed, when/how is it used? As you say, it does not work in this case.
2011-03-05 23:16 UTC
Let's say that you want to change the integer algorithm to return the seed. This can be done like this:

fixture.Customize<int>(c => c.FromSeed(i => i));
2011-03-06 04:24 UTC

Creating Strings With AutoFixture

Thursday, 02 April 2009 05:29:35 UTC

As previously hinted, AutoFixture creates primitive types like strings, numbers, etc. using special algorithms.

In this post, I'll describe the default algorithm for strings. If you don't like this particular algorithm, you can replace it with your own.

If you don't care about the created string at all, you can just create it like this:

string anonymousText = 
    fixture.CreateAnonymous<string>();

The algorithm is simply to create a new Guid and convert it to a string, so that anonymousText will have a value like “f5cdf6b1-a473-410f-95f3-f427f7abb0c7”. Obviously, you don't know exactly which value will be returned, but that's the whole point of Constrained Non-Determinism.

When I create string values as Explicit Expectations, I prefer that the Assert failure message contains some sort of hint for me, so I can instead provide a hint to the CreateAnonymous method:

string anonymousName = fixture.CreateAnonymous("Name");

This overload is still generic, but since I provide a string as input, type inferencing takes care of the rest.

This is still going to create a Guid, but will now prepend the hint, giving a string like “Name30a35da1-d681-441b-9db3-77ff51728b58”.

Now, when my test fails, I'll get an error message equivalent to

"Assert.AreEqual failed. Expected:<Namef2b1f55b-e9dc-4aac-a1ab-128dc80d3b71>. Actual:<ploeh>. Boo hiss"

which I find marginally more informative than if the hint hadn't been there.

In a future post, I'll explain how you can replace this algorithm with something else.


How AutoFixture Creates Objects

Tuesday, 24 March 2009 20:22:49 UTC

AutoFixture creates Anonymous Variables, but you'd probably like to know how it does it. This post explains how.

As we previously saw, the CreateAnonymous method can create a new instance of a type known to it only from its type parameter:

MyClass sut = fixture.CreateAnonymous<MyClass>();

AutoFixture was never compiled with any knowledge of the MyClass type, so it obviously uses Reflection to create the instance. That's hardly surprising in itself.

In the case of MyClass, it has a default constructor, so creating an instance is as simple as it can be, but what happens if we instead ask for a more complex instance?

As an example, the ComplexParent type has this constructor:

public ComplexParent(ComplexChild child)

ComplexChild, however, has two constructors:

public ComplexChild(string name)

and

public ComplexChild(string name, int number)

So what happens when we ask AutoFixture to create an instance of ComplexParent?

ComplexParent only has a single public constructor, so AutoFixture doesn't have any other choice than picking that. This means that it must now create an anonymous instance of ComplexChild.

Fortunately, AutoFixture's raison d'être is creating objects, so creating an instance of ComplexChild isn't a big deal; the only thing it needs to figure out is which constructor to pick. When multiple public constructors are available, it always picks the one with the fewest number of arguments - in this case ComplexChild(string).

Obviously, it then needs to create an anonymous string value. For primitive types like strings, numbers and booleans, AutoFixture has custom algorithms for value creation. Since I'll cover those mechanisms later, suffice it to say that Constrained Non-Determinism is used to create an anonymous string.

At this point, AutoFixture has all the information it needs, and it can now return a properly initialized instance of ComplexParent.

This ability to create instances of almost arbitrarily complex types is a real time-saver: That, more than the ability to create single strings or numbers, was the reason I originally created AutoFixture, since I got tired of initializing complex object graphs just to satisfy some API that the Test Fixture requires.

It also has the additional advantage that it hides all the irrelevant object creation code that the Test Fixture needs, but which isn't relevant for the test at hand.


Comments

That sounds pretty neat. But what if the constructor of ComplexChild imposes some kind of restriction on its parameter? If, for example, instead of the "name" parameter, it would take a "phoneNumber" parameter (as a string), and do some format checking?
2009-03-25 09:17 UTC
Hi Klaus

Thank you for your question! It is an excellent question, particularly because it gives me the opportunity to touch on both some of the more advanced features of AutoFixture, as well as to demonstrate how TDD drives good design.

However, because of that, I'd also like to cover a bit more ground on some of AutoFixture's features before I dive into the details of my answer. In other words, I'd like to post a few more entries that will act as prerequisites before I post my answer to your particular question, so stay tuned, and I'll get to that in due time :)
2009-03-25 13:07 UTC
florin
What if I have two constructors with the same number of parameters, how will it decide on the right one?
2010-05-21 10:51 UTC
It doesn't, so this is a situation best avoided. Essentially AutoFixture orders the constructors according to length, but in case of equal lengths it just picks the first one. Which constructor is the first is up to Reflection to decide, so this is not specified.

This sounds like a serious limitation on AutoFixture's part, but in practice I've found that it rarely matters. In the few cases where a SUT has overloaded constructors, it is essential to the design that none of them puts the SUT into an invalid state, so when you need an anonymous instance it shouldn't matter which constructor is used. This is really what AutoFixture is all about.

That said, in AutoFixture 2.0 it will be possible to register custom types that can use other heuristics to pick constructors.
2010-05-21 12:41 UTC

Joining Safewhere

Monday, 23 March 2009 19:03:38 UTC

Today I spent my first working day at Safewhere, where I'll be working as a Senior Software Engineer. In the last couple of years, Safewhere and I have crossed paths a couple of times, and each time they always left me with the impression of a very professional and congenial ISV, so I'm super-excited to be joining!

Expect the occasional Geneva or Federation-related blog post to intermingle with the usual TDD stuff in the future :)


Comments

Congratulations on the new job :-)
2009-03-24 15:14 UTC
Thanks :)
2009-03-25 07:32 UTC
Mark Rønn
Hi Mark. Great news - congratulations and all the best on your new job :-)
2009-04-15 08:31 UTC

Announcing: AutoFixture

Sunday, 22 March 2009 06:50:54 UTC

It gives me great pleasure to announce my latest project: AutoFixture!

What is AutoFixture?

In essence, AutoFixture is a library that creates Anonymous Variables for you when you write unit tests. The intention is that it should enhance your productivity when you do Test-Driven Development - as it has done mine.

Instead of using mental resources on creating Anonymous Variables, AutoFixture can do it for you. By default, it uses Constrained Non-Determinism, but you can configure it to behave differently if you wish.

Here's a very basic example:

[TestMethod]
public void IntroductoryTest()
{
    // Fixture setup
    Fixture fixture = new Fixture();
 
    int expectedNumber = fixture.CreateAnonymous<int>();
    MyClass sut = fixture.CreateAnonymous<MyClass>();
    // Exercise system
    int result = sut.Echo(expectedNumber);
    // Verify outcome
    Assert.AreEqual<int>(expectedNumber, result, "Echo");
    // Teardown
}

The Fixture class is your main entry point to AutoFixture. You can use it as is, customize it, or derive from it as you please; it makes a great base class for a Fixture Object.

The expectedNumber variable may be an Explicit Expectation, but its value is Anonymous, so instead of coming up with a number ourselves, we can let the CreateAnonymous<T> method do it for us.

This method can create instances of most CLR types as long as they have a public constructor (it uses Reflection), but for many primitive types (like Int32), it has specific, customizable algorithms for creating values using Constrained Non-Determinism.

When creating the SUT, we can also use Fixture as an excellent SUT Factory. Since it will do whatever it can to create an instance of the type you ask for, it is pretty robust if you decide to refactor the SUT's constructor.

The above example only hints at what AutoFixture can do. Since the example is very simple, it may be hard to immediately understand its value, so in future posts I will expand on specific AutoFixture features and principles.

Getting started with AutoFixture is as simple as downloading it from CodePlex and referencing the assembly in Visual Studio.


Comments

Hi Mark,

You could also hook up Pex to AutoFixture (http://research.microsoft.com/pex) and let Pex generate inputs for your tests. Pex provides an API to generate values, similar to your CreateAnonymous method so this should be a piece of cake ;)

Cheers,
Peli
2009-05-01 15:53 UTC
Hi Peli

Thank you for your comment.

Pex is a very interesting piece of technology that I'm very excited about. As I see it, Pex and AutoFixture are complementary technologies. While I've dealt briefly with this topic in the AutoFixture FAQ, it bears repeating here: AutoFixture addresses a different scenario than Pex.

Roughly speaking, Pex is a Quality Assurance (QA) tool - it's main purpose is to break the application, in the time-honored tradition of QA testing. AutoFixture, on the other hand, is a TDD utility library. It's main purpose is to drive development of features along the happy path.

In a full development cycle, I'd use AutoFixture as part of my TDD effort to develop the desired feature. Subsequently, when my API is fairly stable, I'd let Pex loose on it to test its robustness before I ship.

Some day, before version 1.0, I should really let Pex loose on AutoFixture :)
2009-05-01 21:05 UTC

Fixture Object

Monday, 16 March 2009 20:13:29 UTC

(A Zero-Friction TDD post)

For a simple API, setting up the Fixture may be as simple as creating a new instance of the SUT, and possibly any Expected or Anonymous Variables. On the other hand, for a complex API, setting up the fixture may require quite a bit of (potentially repetitive) code.

Since the DRY principle also applies to test code, it quickly becomes necessary to create test-specific helper methods and other SUT API Encapsulation, and I've found that instead of creating a more or less unplanned set of disconnected helper methods, it's much cleaner (and, not to mention, much more object-oriented) to create a single object that represents the Fixture, and attach the helper methods to this object.

Let's look at an example.

Here's a unit test with a complex Fixture Setup:

[TestMethod]
public void NumberSumIsCorrect_Naïve()
{
    // Fixture setup
    Thing thing1 = new Thing()
    {
        Number = 3,
        Text = "Anonymous text 1"
    };
    Thing thing2 = new Thing()
    {
        Number = 6,
        Text = "Anonymous text 2"
    };
    Thing thing3 = new Thing()
    {
        Number = 1,
        Text = "Anonymous text 3"
    };
 
    int expectedSum = new[] { thing1, thing2, thing3 }.
        Select(t => t.Number).Sum();
 
    IMyInterface fake = new FakeMyInterface();
    fake.AddThing(thing1);
    fake.AddThing(thing2);
    fake.AddThing(thing3);
 
    MyClass sut = new MyClass(fake);
    // Exercise system
    int result = sut.CalculateSumOfThings();
    // Verify outcome
    Assert.AreEqual<int>(expectedSum, result,
        "Sum of things");
    // Teardown
}

If this was a truly one-off test and you know with certainty that there's going to be no other tests just remotely similar to this one, just hard-coding the entire Fixture Setup inline is in order, but as soon as the need for similar tests arises, this approach leads to repetitive code, and hence unmaintainable tests.

The more repetitive code that can be delegated to helper methods the better. A common refactoring of the previous test might then look something like this:

[TestMethod]
public void NumberSumIsCorrect_Helpers()
{
    // Fixture setup
    Thing thing1 = MyClassTest.CreateAnonymousThing();
    Thing thing2 = MyClassTest.CreateAnonymousThing();
    Thing thing3 = MyClassTest.CreateAnonymousThing();
    Thing[] things = new[] { thing1, thing2, thing3 };
 
    int expectedSum = things.Select(t => t.Number).Sum();
 
    IMyInterface fake = new FakeMyInterface();
    MyClassTest.AddThingsToMyInterface(fake, things);
 
    MyClass sut = new MyClass(fake);
    // Exercise system
    int result = sut.CalculateSumOfThings();
    // Verify outcome
    Assert.AreEqual<int>(expectedSum, result,
        "Sum of things");
    // Teardown
}

While this is better, the helper methods are static methods, so it's necessary to pass too much state around. The array of Things and the fake is both needed in the test itself, as well as in the AddThingsToMyInterface helper method.

By moving and refactoring the helper methods to a new class that represents the Fixture, the test code becomes both more reusable and more readable.

[TestMethod]
public void NumberSumIsCorrect_FixtureObject()
{
    // Fixture setup
    MyClassFixture fixture = new MyClassFixture();
    fixture.AddAnonymousThings();
 
    int expectedSum = 
        fixture.Things.Select(t => t.Number).Sum();
    MyClass sut = fixture.CreateSut();
    // Exercise system
    int result = sut.CalculateSumOfThings();
    // Verify outcome
    Assert.AreEqual<int>(expectedSum, result,
        "Sum of things");
    // Teardown
}

The MyClassFixture instance now holds the state of the Fixture, so there's much less need to pass around as much data as before. The set of Things is now contained within the Fixture object itself, and the fake has totally disappeared from the test; it's still present, but now encapsulated within MyClassFixture.

internal class MyClassFixture
{
    public MyClassFixture()
    {
        this.Fake = new FakeMyInterface();
        this.Things = new List<Thing>();
    }
 
    internal FakeMyInterface Fake { get; private set; }
 
    internal IList<Thing> Things { get; private set; }
 
    internal void AddAnonymousThings()
    {
        int many = 3;
        for (int i = 0; i < many; i++)
        {
            Thing t = this.CreateAnonymousThing();
            this.Things.Add(t);
            this.Fake.AddThing(t);
        }
    }
 
    internal MyClass CreateSut()
    {
        return new MyClass(this.Fake);
    }
 
    private Thing CreateAnonymousThing()
    {
        Thing t = new Thing();
        t.Number = this.Things.Count + 1;
        t.Text = Guid.NewGuid().ToString();
        return t;
    }
}

The CreateAnonymousThing method uses Constrained Non-Determinism to create unique Thing instances. The AddAnonymousThings method uses 3 as an equivalence of many, and the CreateSut method acts as a SUT Factory.

This is both more reusable and more expressive than a collection of disjointed static helper methods.

Whenever I begin to feel that setting up a Test Fixture is becoming too cumbersome, Fixture Object is the first pattern I consider.


Explicit Expectations

Wednesday, 11 March 2009 19:54:38 UTC

In previous Zero-Friction TDD posts, I've discussed naming SUT and Direct Output variables, as well as the importance of explicitly describing the relationship between input and expected results.

Everything you can do to help the Test Reader understand what's going on increases the quality of the test. Having a naming convention for expectations help in that regard, and it also helps coming up with variable names, thus saving yourself a bit of mental context switching.

My naming convention is to always prefix my expectation variables with the term expected.

[TestMethod]
public void DoStuffWillReturnMessage()
{
    // Fixture setup
    string expectedResult = "ploeh";
    MyClass sut = new MyClass();
    // Exercise system
    string result = sut.DoStuff(expectedResult);
    // Verify outcome
    Assert.AreEqual<string>(expectedResult, result,
        "DoStuff");
    // Teardown
}

In a test like this, the relationship between the input and output is straightforward, but in other cases it may be more complicated. Since tests should explicitly state the relationship between input and output, it may sometimes be necessary to reproduce parts of the SUT's behavior in the test to specify this association.

Do I really recommend duplicating the SUT's code in the test? Isn't this a violation of the DRY principle? And do I really think that embedding complex code in a test is a good idea?

No, no, and no.

What I really mean is best illustrated with an example. Imagine that you want to write an extension method that converts a string to PascalCase. There are several different rules that must be applied to such an algorithm, such as

  • Convert the first letter in a word to upper case
  • Convert the remaining letters in the word to lower case
  • Remove white space

The real algorithm would need to split the string into words along white space boundaries, then loop through this list and perform the conversion for each word, and finally concatenate all the words. However, I don't think you should reproduce this algorithm in any single test.

What you can do instead is to split this behavior into several tests that each test a small part of this specification, carefully avoiding any control flow language features (such as if, switch, for, etc.).

One such test might look like this:

[TestMethod]
public void ToPascalCaseWillConvertFirstLetterToUpper()
{
    // Fixture setup
    string anonymousText = "pLOeh";
    string expectedLetter =
        anonymousText.First().ToString().ToUpper();
    // Exercise system
    string result =
        anonymousText.ToPascalCase();
    // Verify outcome
    Assert.AreEqual<string>(expectedLetter,
        result.First().ToString(), "ToPascalCase");
    // Teardown
}

While the complete implementation of ToPascalCase is more complex, I've extracted a tiny bit of the specification and simulated just that for the special case where there's only one word. Granted, there's a lot of method calls, but I expect that these have already been thoroughly tested, so I use them with confidence. The cyclomatic complexity of the test is minimal.

As an aside, note that I'm using LINQ queries to get the first letter of the string, instead of Substring(0, 1), since I find that the LINQ methods much better communicate intent.

I use a similar naming convention for unexpected values, imaginatively prefixing my variables with the term unexpected.

[TestMethod]
public void CreateThingWillCreateThingWithCorrectGuid()
{
    // Fixture setup
    Guid unexpectedId = Guid.Empty;
    MyClass sut = new MyClass();
    // Exercise system
    Thing result = sut.CreateThing();
    // Verify outcome
    Assert.AreNotEqual<Guid>(unexpectedId, result.Id,
        "CreateThing");
    // Teardown
}

Having a naming convention for expected values not only increases your productivity when writing tests, but also increases test maintainability.


Comments

Hi,

I've read this post a while back, and it was a really helpful way for me to see you suggestion how to still use the "explicit relationship" between input, and output, but preventing yourself from reproducing the implementation by breaking the various parts of the algorithm apart.
I've used that already in plenty of situations, where I would have written a single unit test which was testing multiple scenario's, which could be split up in various isolated unit tests.
However, the other day, I needed to write a view model property that combines 2 properties of that view model in a particular format (see below for the code). I also tried to use this approach, but I couldn't find a way to do that. How would you break that apart in different unit tests?
public string NameAndType { get { return string.Format("{0} [{1}]", Name, ContracteeType); } }
2015-02-3 15:33 UTC

Like this?

[Theory]
[InlineData("Foo""Bar""Foo [Bar]")]
[InlineData("Baz""Qux""Baz [Qux]")]
public void NameAndTypeReturnsCorrectResult(
    string name,
    string contracteeType,
    string expected)
{
    var sut = new MyClass(name, contracteeType);
    var actual = sut.NameAndType;
    Assert.Equal(expected, actual);
}

I'm not sure I understand the question...

2015-02-03 17:03 UTC

Specification-Driven Development

Tuesday, 10 March 2009 21:04:38 UTC

In the last couple of years, there's been a lot of debate in the community on the philosophy behind TDD and where to put the emphasis - even to the point of debating whether the acronym stands for Test-Driven Development or Test-Driven Design.

Other people don't like the emphasis on tests, since that makes TDD sound like a Testing discipline, and not a Development discipline. Instead, they prefer terms like Example-Driven Design/Development (EDD) or even Design by Example (DbE).

This view seems to me to be particularly prevalent in Microsoft, where there's a rather sharp distinction between developers and testers (job titles to the contrary) - I guess that's one of the reasons why xUnit.net (a project initiated by Microsoft employees) uses the attribute Fact instead of Test or TestMethod.

For people used to SCRUM or other agile methodologies, this distinction is more blurred, and they also seem to accept the T in TDD more willingly.

However, the adherents of EDD claim that the mere presence of the word test make some developers block any further input and stop listening. They may be right in that.

They also claim that the tests in TDD/EDD are nothing more than accidental artifacts of the development process, and hence argue that we shouldn't call them tests at all. However, if that's true, this little story related by Ayende must be an example of EDD in its purest form :)

To me, the tests are also important. Since 2003 I've been practicing TDD, and while I love how it helps me arrive at better design, I also savor the safety net that my suite of tests gives me. The tests that I write during TDD define the behavior of the software. In many cases, I'd even claim that such a regression test suite is more valuable than a Quality Assurance (QA) regression test suite - after all, a QA suite may catch some edge cases, but they don't focus on the intended behavior of the system, but often more on how to break it - but I digress…

My recent posts on Executable Specification and Constrained Non-Determinism help explain my current stance in this debate: In my opinion, EDD fails to establish a relationship by not providing Derived Values. After all, what does a test like the following specify?

[TestMethod]
public void InvertWillReverseText_Naïve()
{
    // Fixture setup
    MyClass sut = new MyClass();
    // Exercise system
    string result = sut.Invert("ploeh");
    // Verify outcome
    Assert.AreEqual<string>("heolp", result, "Invert");
    // Teardown
}

How would you implement the Invert method? Here's one possible implementation:

return "heolp";

Obviously, you could now write a new test that gives a second example of input and outcome and force me to write a more sophisticated algorithm. However, with only two examples, I might still be tempted to write a switch statement with some hard-coded return values until you've written so many ‘examples' that you've coerced me into writing the more general (and correct) algorithm.

Such an approach I find inefficient.

Instead, by using Constrained Non-Determinism to force myself to define Derived Values, each test fully specifies the desired behavior. It doesn't provide examples. It provides the specification, and instead of having to write several similar examples to coerce a general algorithm to emerge, I can usually nail it in a single test.

This approach could be styled Specification-Driven Development, and that's how I've been writing code for the last year or so.


Comments

Just a comment... Remember the pattern in TDD: Red, Green, Refactor. You're not supposed to write a second test in order to force you to remove the hardcoded return "heolp"; implementation. You're supposed to remove duplication during the refactoring stage. The constant "heolp" is duplicated in the test and in the implementation code, and according to Kent Beck this is your main focus during refactoring; to eliminate duplication.
2009-03-28 12:28 UTC
Hi Torbjørn

Thank you for your comment.

First of all: I'm a big believer in the Red/Green/Refactor cycle, so we're totally on the same page there!

As the test in the example stands, the constant "heolp" is most certainly duplicated across test and implementation code, until the point when I decide to implement the Invert method correctly. So far I can only agree, but partially because my example is so simple: Once again I fall into the trap that my example is too simple, but had I made it more complex, I'd have lost my reader(s) long ago.

One or few 'examples' would be enough to clue anyone on to the fact that the intended algorithm of the Invert method is to invert the input. It's fairly obvious.

Imagine, instead, that we're testing a hypothetical CalculateRocketTrajectory method. Yes, we're attempting to implement a Domain Model over rocket science! Unless you are a rocket scientist, it's likely that it's going to take a pretty big amount of 'examples' before you can figure out what is the underlying algorithm. It would be much more efficient if each test could simply specify the relationship between input and output.

Yes, that would lead to a bit of duplication across the test code and the implementation code, but I prefer that to the alternative. In my experience, most software development isn't about algorithm development anyway; it's about API design, Domain Modeling, handling corner cases, moving and transforming data, etc.

This is not to say that I am right and you are wrong. Here, I'm mainly trying to describe what motivates me to work in a certain way, so please feel free to keep the discussion going!
2009-03-28 13:55 UTC
I expect you have much more actual experience with TDD (or EDD or DbE or BDD) then myself. Since I just read Beck's TDD book and had it fresh in mind, I just wanted to give my comment, for your readers mostly, since I felt that part of the pattern wasn't represented in your post.

Keep up the nice posts.., enjoying your blog!
2009-03-28 18:01 UTC
Hi Torbjørn

Thanks again. You point was valid and well taken.

One of the reasons I find EDD ineffecient stems from my experience with The XP Game, which is enjoyable, but a bit too much on the back-and-forth side for my tastes.

YMMV, so I'm not trying to dictate what anyone should do - I'm just trying to demonstrate that there are more than one way to do TDD.
2009-03-28 19:54 UTC

Constrained Non-Determinism

Thursday, 05 March 2009 20:23:05 UTC

This may turn out to be the most controversial of my Zero-Friction TDD posts so far, as it supposedly goes against conventional wisdom. However, I have found this approach to be really powerful since I began using it about a year ago.

In my previous post, I explained how Derived Values help ensure that tests act as Executable Specification. In short, a test should clearly specify the relationship between input and outcome, as this test does:

[TestMethod]
public void InvertWillReverseText()
{
    // Fixture setup
    string anonymousText = "ploeh";
    string expectedResult =
        new string(anonymousText.Reverse().ToArray());
    MyClass sut = new MyClass();
    // Exercise system
    string result = sut.Invert(anonymousText);
    // Verify outcome
    Assert.AreEqual<string>(expectedResult, result,
        "DoWork");
    // Teardown
}

However, it is very tempting to just hardcode the expected value. Consistently using Derived Values to establish the relationship between input and outcome requires discipline.

To help myself enforce this discipline, I use well-defined, but essentially random, input, because when the input is random, I don't know the value at design time, and hence, it is impossible for me to accidentally hard-code any assertions.

[TestMethod]
public void InvertWillReverseText_Cnd()
{
    // Fixture setup
    string anonymousText = Guid.NewGuid().ToString();
    string expectedResult =
        new string(anonymousText.Reverse().ToArray());
    MyClass sut = new MyClass();
    // Exercise system
    string result = sut.Invert(anonymousText);
    // Verify outcome
    Assert.AreEqual<string>(expectedResult, result,
        "DoWork");
    // Teardown
}

For strings, I prefer Guids, as the above example demonstrates. For numbers, I often just use the sequence of natural numbers (i.e. 1, 2, 3, 4, 5...). For booleans, I often use an alternating sequence (i.e. true, false, true, false...).

While this technique causes input to become non-deterministic, I always pick the non-deterministic value-generating algorithm in such a way that it creates 'nice' values; I call this principle Constrained Non-Determinism. Values are carefully generated to stay far away from any boundary conditions that may cause the SUT to behave differently in each test run.

Conventional unit testing wisdom dictates that unit tests should be deterministic, so how can I possibly endorse this technique?

To understand this, it's important to know why the rule about deterministic unit tests exist. It exists because we want to be certain that each time we execute a test suite, we verify the exact same behavior as we did the last time (given that no tests changed). Since we also use test suites as regression tests, it's important that we can be confident that each and every test run verifies the exact same specification.

Constrained Non-Determinism doesn't invalidate that goal, because the algorithm that generates the values must be carefully picked to always create values that stay within the input's Equivalence Class.

In a surprisingly large set of APIs, strings, for example, are treated as opaque values that don't influence behavior in themselves. Many enterprise applications mostly store and read data from persistent data stores, and the value of a string in itself is often inconsequential from the point of view of the code's execution path. Data stores may have constraints on the length of strings, so Constrained Non-Determinism dictates that you should pick the generating algorithm so that the string length always stays within (or exceeds, if that's what you want to test) the constraint. Guid.ToString always returns a string with the length of 36, which is fine for a large number of scenarios.

Note that Constrained Non-Determinism is only relevant for Anonymous Variables. For input where the value holds a particular meaning in the context of the SUT, you will still need to hand-pick values as always. E.g. if the input is expected to be an XML string conforming to a particular schema, a Guid string makes no sense.

A secondary benefit of Constrained Non-Determinism is that you don't have to pause to come up with values for Anonymous Variables when you are writing the test.

While this advice may be controversial, I can only recommend it - I've been using this technique for about a year now, and have only become more fond of it as I have gained more experience with it.


Comments

Hey Phoeh, this is @CarlJ (if you didn't guess).

One suggestion/tip when using random values to test, is to output the random value when the test fails. With randomness, you may come across that 1 in 100 value that breaks the test, but you won't have any idea of what it was, which makes it a bit hard to replicate and fix. In my Unit Test project, I have a common method that you pass the value to and it will print it to the output screen when it fails.

As for my question on Twitter. I'm dealing with a really large database (700Gbs), which returns an infinite number of combinations of data through stored procs. There are no INSERT/UPDATE/DELETEs (that's done in a totally different project). What I want to test is how the code handles the data that is returned from the procs, which is based on a user's selection from multiple radio buttons, and drop down boxes (single and multiple selection).

This data that they select from, comes from the database too, which we have little control over if it's valid or not.

So my question on Twitter was, should I create a method(s) that generate random parameters (which come from the DB) that the proc accepts? What I'm testing is not if the data itself is valid, but whether the code handles some weird anomalies? Or should I just use parameters that I know return valid data?

I've already experimented with creating a proc that generates random parameters from the data within the database, and using those values in my test. Amazingly, it's found a lot of issues that I was able to go back and fix.

The reason why I ask, is because I've heard that this goes beyond "Unit Testing".

Thanks,
Carl J
2010-10-08 14:52 UTC
It sounds to me like Pex would fit your scenario very well. Basically, that would allow you to write a Pex test that creates deterministic test cases for each and every code path through the SUT. I do realize that data comes from the database, but it's still input. In testing terminology, we call that Indirect Input.

If you can't inject a Test Double that provides all the different data combinations, you should be able to use Moles for that part of the task.
2010-10-08 15:57 UTC

Page 38 of 39

"Our team wholeheartedly endorses Mark. His expert service provides tremendous value."
Hire me!