Rapid feedback is one of the cornerstones of agile development. The faster we can get feedback, the less it costs to correct errors. Unit testing is one of the ways to get feedback, but not the only one. Each way we can get feedback comes with its own advantages and disadvantages.

In my experience there's a tradeoff between the cost of setting up a feedback mechanism and the level of confidence we can get from it. This is quite intuitive to most people, but I've rarely seen it explicitly described. The purpose of this post is to explicitly describe and relate those different mechanisms.

Compilation

In compiled languages, compilation provides the first level of feedback. I think a lot of people don't think about this as feedback as (for compiled languages) it's a necessary step before the code can be executed.

The level of confidence may not be very high - we tend to laugh at statements like ‘if it compiles, it works'. However, the compiler still catches a lot of mistakes. Think about it: does your code always compile, or do you sometimes get compilation errors? Do you sometimes try to compile your code just to see if it's possible? I certainly do, and that's because the compiler is always available, and it's the first and fastest level of feedback we get.

Code can be designed to take advantage of this verification step. Constructor Injection, for example, ensures that we can't create a new instance of a class without supplying it with its required dependencies.

It's also interesting to note that anecdotal evidence seems to suggest that unit testing is more prevalent for interpreted languages. That makes a lot of sense, because as developers we need feedback as soon as possible, and if there's no compiler we must reach for the next level of feedback mechanism.

Static code analysis

The idea of static code analysis isn't specific to .NET, but in certain versions of Visual Studio we have Code Analysis (which is also available as FxCop). Static code analysis is a step up from compilation - not only is code checked for syntactical correctness, but it's also checked against a set of known anti-patterns and idioms.

Static code analysis is encapsulated expert knowledge, yet surprisingly few people use it. I think part of the reason for this is because the ratio of time against confidence isn't as compelling as with other feedback mechanism. It takes some time to run the analysis, but like compilation the level of confidence gained from getting a clean result is not very high.

Personally, I use static code analysis once in a while (e.g. before checking in code to the default branch), but not as often as other feedback mechanisms. Still, I think this type of feedback mechanism is under-utilized.

Unit testing

Running a unit test suite is one of the most efficient ways to gain rapid feedback. The execution time is often comparable to running static code analysis while the level of confidence is much higher.

Obviously a successful unit test execution is no guarantee that the final application works as intended, but it's a much better indicator than the two previous mechanisms. When presented with the choice of using either static code analysis or unit tests, I prefer unit tests every time because the confidence level is much higher.

If you have sufficiently high code coverage I'd say that a successful unit test suite is enough confidence to check in code and let continuous integration take care of the rest.

Keep in mind that continuous integration and other types of automated builds are feedback mechanisms. If you institute rules where people are ‘punished' for checking in code that breaks the build, you effectively disable the automated build as a source of feedback.

Thus, the rest of these feedback mechanisms should be used rarely (if at all) on developer work stations.

Integration testing

Integration testing comes in several sub-categories. You can integrate multiple classes from within the same library, or integrate multiple libraries while still using Test Doubles instead of out-of-process resources. At this level, the cost/confidence ratio is comparable to unit testing.

Subcutaneous testing

A more full level of integration testing comes when all resources are integrated, but the tests are still driven by code instead of through the UI. While this gives a degree of confidence that is close to what you can get from a full systems test, the cost of setting up the entire testing environment is much higher. In many cases I consider this the realm of a dedicated testing department, as it's often a full-time job to maintain the suite of automation tools that enable such tests - particularly if we are talking about distributed systems.

System testing

This is a test of the full system, often driven through the UI and supplemented with manual testing (such as exploratory testing). This provides the highest degree of confidence, but comes with a very high cost. It's often at this level we find formal acceptance tests.

The time before we can get feedback at this level is often considerable. It may take days or even weeks or months.

Understand the cost/confidence ratio

The purpose of this post has been to talk about different ways we can get feedback when developing software. If we could get instantaneous feedback with guaranteed confidence it would be easy to choose, but as this is not the case we must understand the benefits and drawbacks of each mechanism.

It's important to realize that there are many mechanisms, and that we can combine them in a staggered approach where we start with fast feedback (like the compiler) with a low degree of confidence and then move through successive stages that each provide a higher level of confidence.


Comments

Markus Hjärne
Hi Mark, good post. I just thought of one thing worth mentioning in this context: guarding public APIs with checks that throws exceptions in case they fail and guarding private methods with asserts has been a great source of good and fast feedback for me over the years.

Regards,
Markus
2011-05-04 20:46 UTC
I totally agree, and I was (and am still) planning on writing something about that in a later post. This is essentially the Fail Fast principle, but as for feedback mechanisms, we don't discover these Guard Clauses before we invoke them.

My strong preference is for writing unit tests that invoke these Guard Clauses to verify that they work, and that means that as far as feedback mechanisms go, they fall into the unit testing 'bucket'.
2011-05-04 20:54 UTC
Your comments about compilation are interesting, I hadn't thought of it before but it is a level of feedback that I find as useful as you do.

I wonder if you'd consider IntelliSense as an even earlier form of feedback though?

Gog
2012-06-25 09:20 UTC
IntelliSense is, I think, more of a learning tool. It's not really feedback about what you did - it's more information about what you can do. It's certainly a productivity feature, although there are some people who argue that it does more harm than good.
2012-06-25 10:50 UTC
It might be worth considering a couple of additional feedback mechanisms at the opposite end of the spectrum, in the sense that they provide even later feedback, if only to underscore your points from the article with extreme counter-examples. Run-time exceptions are themselves a feedback mechanism. Most unnecessarily "stringly-typed" code could be considered to be pushing code away from the more rapid, desirable feedback mechanisms. (Primitive Obsession in general might be considered another tendency which pushes the design of code toward being run-time exception-prone.) Logic erors are flaws in code with which the code appears to execute successfully, until the program output is manually inspected. Just off the cuff, one typical cause might be the tendency in some projects to add guard clauses such as if (methodArg==null) return; which causes methods to silently and unexpectedly return without accomplishing the task that they were designed to do. Mathematical calculations performed without adequate test coverage might be another example.

We learned about 3 classes of errors in school: 1) Syntax, 2) Run-time, and 3) Logic (Unit testing and static code analysis weren't as much of a thing back then). It might seem odd to discuss types of feedback mechanisms that are undesirable by definition, until you stop to observe how common it is for the design of much code to push errors towards this undesirable side of the continuum. "To defeat your enemy you first must know him" or something like that.
2016-09-11 00:11 UTC

David, thank you for making these feedback cases explicit. When I originally wrote this article, I implicitly had such alternative, common-place feedback mechanisms in mind, but I never explicitly spelled them out. While I should have done that, I can now thank you for doing it; your comment is a good addition to the original article.

2016-09-11 08:07 UTC


Wish to comment?

You can add a comment to this post by sending me a pull request. Alternatively, you can discuss this post on Twitter or Google Plus, or somewhere else with a permalink. Ping me with the link, and I may add it as a comment.

Published

Friday, 29 April 2011 13:02:14 UTC

Tags



"Our team wholeheartedly endorses Mark. His expert service provides tremendous value."
Hire me!