About a month ago I decided to migrate from MSTest to xUnit.net, and while I am still in the process, I haven't regretted it yet, and I don't expect to. AutoFixture has already moved over, and I'm slowly migrating all the sample code for my book.

Recently I was asked why, which prompted me to write this post.

I'm not moving away from MSTest for one single reason. It's rather like lots of small reasons.

When I originally started out with TDD, I used nUnit - it was more or less the only unit testing framework available for .NET at the time. When MSTest came, the change was natural, since I worked for Microsoft at the time. This is not the case anymore, but it still took me most of a year to finally abandon MSTest.

There was one thing that really made me cling to MSTest, and that was the IDE integration, but over time, I started to realize that this was the only reason, and even that was getting flaky.

When I started to think about all the things that left me dissatisfied, making the decision was easy:

  • First of all, MSTest isn't extensible, but xUnit.net is. In xUnit.net, I can extend the Fact or Theory attributes (and I intent to), while in MSTest, I will have to play with the cards I've been dealt. I think I could live with all the other issues if I could just have this one, but no.
  • MSTest has no support for parameterized test. xUnit.net does (via the Theory attribute).
  • MSTest has no Assert.Throws, although I requested this feature a long time ago. Now Visual Studio 2010 is out, but Assert.Throws is still nowhere in sight.
  • MSTest has no x64 support. Tests always run as x86. Usually it's no big deal, but sometimes it's a really big deal.
  • In MSTest, to write unit tests, you must create a special Unit Test Project, and those are only available for C# and VB.net. Good luck trying to write unit tests in a more exotic .NET language (like F# on Visual Studio 2008). xUnit.net doesn't have this problem.
  • MSTest uses Test Lists and .vsmdi files to maintain test lists. Why? I don't care, I just want to execute my tests, and the .vsmdi files are in the way. This is particularly bad when you use TFS, but I'm also moving away from TFS, so that wouldn't have continued to be that much of an issue. Still: try having more than one .sln file with unit tests in the same folder, and watch funny things happen because they need to share the same .vsmdi file.
  • I suppose it's because of the .vsmdi files, but sometimes I get a Test run error if I delete a test and run the tests immediately after. That's a false positive, if anyone cares.
  • MSTest gives special treatment to its own AssertionException, which gets nice formatting in the Test Results window. All other exceptions (like verification exceptions thrown by Moq or Rhino Mocks are rendered near-intelligible because MSTest thinks it's very important to report the fully qualified name of the exception before its message. Most of the time, you have to open the Test Details window to see the exception message.
  • Last, but not least, I often get cryptic exception messages like this one: Column 'id_column, runid_column' is constrained to be unique.  Value '8c84fa94-04c1-424b-9868-57a2d4851a1d, d7471c5e-522f-43d3-b2c5-8f5cab55af0e' is already present. This appears in a very annoying modal MessageBox, but clicking OK and retrying usually works, although sometimes it even takes two or three attempts before I can get past this error.

It not one big thing, it's just a lot of small, but very annoying things. After three iterations (VS2005, VS2008 and now VS2010) these issue have still to be addressed, and I got tired of waiting.

So far, I can only say that I have none of these problems with xUnit.net and the IDE integration provided by TestDriven.NET. It's just a much smoother experience with much less friction.


Comments

Hi Mark,

Interesting post about moving away from "out-of-the-box" Microsoft tools. I've made this move about a year ago, and I can't regret about it.

Another point that you mentioned in your post and that really caught my attention was the fact that you are also moving away from TFS. Since I'm starting my own startup here, the budget is really short and we are looking for cheaper alternatives to TFS.

Here, we really like Mercurial HG and we are basing out SCM on it. However, I'm having difficulty finding tools for bug and feature tracking. Can you share with me in which direction you are moving away from TFS?
2010-04-26 13:59 UTC
Personally, I also use Hg for SCM.

In Safewhere, we are currenlty trying out AgileZen for work item tracking. For AutoFixture, I just use the tools provided with CodePlex.
2010-04-26 16:32 UTC
I started with NUnit, and gave a quick shot at MSTest, but never made the transition, because I couldn't see any upside. Could you comment a bit on why you picked xUnit over NUnit? I haven't tried it yet, but from what I saw in the docs, the syntax is pretty interesting; first time in a while that I see a framework which seems to re-think unit testing, rather than improve on JUnit.
2010-04-26 21:15 UTC
There are two main reasons that I prefer xUnit.net over NUnit, but both may be due to ignorance about NUnit on my part. The last time I did serious work with NUnit must have been back in 2005.

One reason is that xUnit.net has a pretty good extensibility story, and as I do have some plans in that direction, that's a pretty big issue for me. Last time I checked, the extensibility story for NUnit didn't match xUnit.net.

NUnit has a design bug when it comes to Fixture management, because it creates only a single instance of a test class and invokes all the test methods on that instance. This may have been fixed since the last time I looked, but in any case, I better like xUnit.net's philosphy of using the test class' constructor for Fixture Setup (if any) and implementing IDisposable for Fixture Teardown.

As I said, both items are based on somewhat dated knowledge on my part, so none of them may apply anymore.
2010-04-26 21:26 UTC
Thanks for writing this up, the tooling built into VS for MSTest makes it very attractive but it still gets a lot of hate. Nice to know why :)
2010-05-11 10:36 UTC
Is the MSTest code going to also be available somewhere? I'd hate to have to constantly be translating for code you already have available for us. - twitter @MaslowJax
2010-06-19 16:46 UTC
I'm not sure exactly to which MSTest code you are referring, but in general I don't plan to change the existing MSTest code I've posted here on the blog. However, new tests are likely to appear with xUnit.net. In any case, when it comes to unit testing examples I don't think the differences are all that important. In most cases it's just a question of differently named attributes and slightly different Assert syntax...
2010-06-20 06:39 UTC
Fabricio
MSTest support parametrized tests using Pex
2012-03-27 15:17 UTC


Wish to comment?

You can add a comment to this post by sending me a pull request. Alternatively, you can discuss this post on Twitter or Google Plus, or somewhere else with a permalink. Ping me with the link, and I may add it as a comment.

Published

Monday, 26 April 2010 04:30:49 UTC

Tags



"Our team wholeheartedly endorses Mark. His expert service provides tremendous value."
Hire me!