Where to put unit tests by Mark Seemann

This article provides arguments for (and against) putting unit tests in libraries different from the production code.

One of my readers ask me "whether unit tests should be done in separate assemblies or if they should be done in [...] the production assemblies?" As he puts it, he "always took for granted that the test should go into separate assemblies," but is this really true, and if it is: what are the arguments?

Despite the fundamental nature of this question, I haven't seen much explicit treatment of it, and to answer it, I had to pause and think. The key to the answer, I think, is to first understand why you write automated tests at all. (This way of thinking about questions and answers is closely related to the (Lean) technique of 5 whys.)

As always, the short answer is that it depends, but that's not very helpful, so here are some common reasons.

Regression testing #

In my experience, most people focus on the quality control aspect of automated testing. If your only purpose of having automated tests is to have a good regression test suite, then it seems that it doesn't really matter where the tests go.

Still, even if that's your only reason for unit testing, I think putting the unit tests in a separate library is the best choice:

• It gives you a clearer separation of concerns. If you put unit tests in your production library, it will blur the distinction between production code and test code. How do you know which code to test? How do you know what not to test? There are different ways to address these concerns (namespaces, TDD), but I think the simplest way to deal with such issues is to put the tests in a separate library. (This argument reminds me a little of Occam's razor.)
• Putting unit tests in your production code will make your compiled libraries bigger. It will take (slightly) more time to load the libraries into memory, and they will take up more memory. This probably doesn't matter at all on a big server, but may be important if your production code runs on a (small) mobile device. Once again, it depends.
• The more code you put in your production software, the larger the potential attack surface becomes. Have you performed a security analysis of your unit tests? It's much easier to put the tests in a separate library that you never deploy to production, because it means that you don't have to waste valuable time and resources doing a security analysis of your test code.

In short, if you put unit tests in the same library as your production code, all that unit test must adhere to the same standards as your production code. Obviously, if you have no standards for your production code, this doesn't apply, but then you probably have bigger problems.

API design feedback #

While I find regression testing useful in itself, I find Test-Driven Development (TDD) particularly valuable because it provides feedback about the API I'm building. In the spirit of GOOS, if the test is difficult to write, it's probably because the API is difficult to use. You'll only get this feedback if you test against the public API of your code; the unit test is the first client of your API.

However, if your unit tests reside in the same library as your production code, you can easily invoke internal classes and members from your unit tests. You simply lose the opportunity to get valuable feedback about the usability of your code. (Note that this argument also applies to the use of the [InternalsVisibleTo] attribute, which means that you should never use it.)

For me, this reason alone is enough that I always prefer putting unit tests in separate libraries.

Shipping production code with tests #

Although I think that there are strong arguments against putting unit tests in production code, there may also be reasons in favor. As one answer on Stack Overflow describes, shipping your production code with a test suite can be valuable as a self-diagnostics tool. This might be particularly valuable if you deploy your production code to heterogeneous environments, and you're unable to predict and test the configuration of all environments before you ship your code.

Summary #

In the end, the answer depends on your context. If you understand why you (want to) have unit tests, you'll be able to answer the initial question: should unit tests go in a separate library, or can they go in the same library as the production code? In mainstream scenarios I will strongly recommend putting tests in a separate library, but you may have special circumstances where it makes sense to put the tests together with the production code. As long as you understand the pros and cons, you can make your own informed decision.

Comments

• ← Previous
• Archive
• Next →