Why I'm migrating from MSTest to xUnit.net by Mark Seemann
About a month ago I decided to migrate from MSTest to xUnit.net, and while I am still in the process, I haven't regretted it yet, and I don't expect to. AutoFixture has already moved over, and I'm slowly migrating all the sample code for my book.
Recently I was asked why, which prompted me to write this post.
I'm not moving away from MSTest for one single reason. It's rather like lots of small reasons.
When I originally started out with TDD, I used nUnit - it was more or less the only unit testing framework available for .NET at the time. When MSTest came, the change was natural, since I worked for Microsoft at the time. This is not the case anymore, but it still took me most of a year to finally abandon MSTest.
There was one thing that really made me cling to MSTest, and that was the IDE integration, but over time, I started to realize that this was the only reason, and even that was getting flaky.
When I started to think about all the things that left me dissatisfied, making the decision was easy:
- First of all, MSTest isn't extensible, but xUnit.net is. In xUnit.net, I can extend the Fact or Theory attributes (and I intent to), while in MSTest, I will have to play with the cards I've been dealt. I think I could live with all the other issues if I could just have this one, but no.
- MSTest has no support for parameterized test. xUnit.net does (via the Theory attribute).
- MSTest has no Assert.Throws, although I requested this feature a long time ago. Now Visual Studio 2010 is out, but Assert.Throws is still nowhere in sight.
- MSTest has no x64 support. Tests always run as x86. Usually it's no big deal, but sometimes it's a really big deal.
- In MSTest, to write unit tests, you must create a special Unit Test Project, and those are only available for C# and VB.net. Good luck trying to write unit tests in a more exotic .NET language (like F# on Visual Studio 2008). xUnit.net doesn't have this problem.
- MSTest uses Test Lists and .vsmdi files to maintain test lists. Why? I don't care, I just want to execute my tests, and the .vsmdi files are in the way. This is particularly bad when you use TFS, but I'm also moving away from TFS, so that wouldn't have continued to be that much of an issue. Still: try having more than one .sln file with unit tests in the same folder, and watch funny things happen because they need to share the same .vsmdi file.
- I suppose it's because of the .vsmdi files, but sometimes I get a Test run error if I delete a test and run the tests immediately after. That's a false positive, if anyone cares.
- MSTest gives special treatment to its own AssertionException, which gets nice formatting in the Test Results window. All other exceptions (like verification exceptions thrown by Moq or Rhino Mocks are rendered near-intelligible because MSTest thinks it's very important to report the fully qualified name of the exception before its message. Most of the time, you have to open the Test Details window to see the exception message.
- Last, but not least, I often get cryptic exception messages like this one: Column 'id_column, runid_column' is constrained to be unique. Value '8c84fa94-04c1-424b-9868-57a2d4851a1d, d7471c5e-522f-43d3-b2c5-8f5cab55af0e' is already present. This appears in a very annoying modal MessageBox, but clicking OK and retrying usually works, although sometimes it even takes two or three attempts before I can get past this error.
It not one big thing, it's just a lot of small, but very annoying things. After three iterations (VS2005, VS2008 and now VS2010) these issue have still to be addressed, and I got tired of waiting.
So far, I can only say that I have none of these problems with xUnit.net and the IDE integration provided by TestDriven.NET. It's just a much smoother experience with much less friction.
Comments
Interesting post about moving away from "out-of-the-box" Microsoft tools. I've made this move about a year ago, and I can't regret about it.
Another point that you mentioned in your post and that really caught my attention was the fact that you are also moving away from TFS. Since I'm starting my own startup here, the budget is really short and we are looking for cheaper alternatives to TFS.
Here, we really like Mercurial HG and we are basing out SCM on it. However, I'm having difficulty finding tools for bug and feature tracking. Can you share with me in which direction you are moving away from TFS?
In Safewhere, we are currenlty trying out AgileZen for work item tracking. For AutoFixture, I just use the tools provided with CodePlex.
One reason is that xUnit.net has a pretty good extensibility story, and as I do have some plans in that direction, that's a pretty big issue for me. Last time I checked, the extensibility story for NUnit didn't match xUnit.net.
NUnit has a design bug when it comes to Fixture management, because it creates only a single instance of a test class and invokes all the test methods on that instance. This may have been fixed since the last time I looked, but in any case, I better like xUnit.net's philosphy of using the test class' constructor for Fixture Setup (if any) and implementing IDisposable for Fixture Teardown.
As I said, both items are based on somewhat dated knowledge on my part, so none of them may apply anymore.