Feedback mechanisms and tradeoffs by Mark Seemann

Rapid feedback is one of the cornerstones of agile development. The faster we can get feedback, the less it costs to correct errors. Unit testing is one of the ways to get feedback, but not the only one. Each way we can get feedback comes with its own advantages and disadvantages.

In my experience there's a tradeoff between the cost of setting up a feedback mechanism and the level of confidence we can get from it. This is quite intuitive to most people, but I've rarely seen it explicitly described. The purpose of this post is to explicitly describe and relate those different mechanisms.

Compilation #

In compiled languages, compilation provides the first level of feedback. I think a lot of people don't think about this as feedback as (for compiled languages) it's a necessary step before the code can be executed.

The level of confidence may not be very high - we tend to laugh at statements like ‘if it compiles, it works'. However, the compiler still catches a lot of mistakes. Think about it: does your code always compile, or do you sometimes get compilation errors? Do you sometimes try to compile your code just to see if it's possible? I certainly do, and that's because the compiler is always available, and it's the first and fastest level of feedback we get.

Code can be designed to take advantage of this verification step. Constructor Injection, for example, ensures that we can't create a new instance of a class without supplying it with its required dependencies.

It's also interesting to note that anecdotal evidence seems to suggest that unit testing is more prevalent for interpreted languages. That makes a lot of sense, because as developers we need feedback as soon as possible, and if there's no compiler we must reach for the next level of feedback mechanism.

Static code analysis #

The idea of static code analysis isn't specific to .NET, but in certain versions of Visual Studio we have Code Analysis (which is also available as FxCop). Static code analysis is a step up from compilation - not only is code checked for syntactical correctness, but it's also checked against a set of known anti-patterns and idioms.

Static code analysis is encapsulated expert knowledge, yet surprisingly few people use it. I think part of the reason for this is because the ratio of time against confidence isn't as compelling as with other feedback mechanism. It takes some time to run the analysis, but like compilation the level of confidence gained from getting a clean result is not very high.

Personally, I use static code analysis once in a while (e.g. before checking in code to the default branch), but not as often as other feedback mechanisms. Still, I think this type of feedback mechanism is under-utilized.

Unit testing #

Running a unit test suite is one of the most efficient ways to gain rapid feedback. The execution time is often comparable to running static code analysis while the level of confidence is much higher.

Obviously a successful unit test execution is no guarantee that the final application works as intended, but it's a much better indicator than the two previous mechanisms. When presented with the choice of using either static code analysis or unit tests, I prefer unit tests every time because the confidence level is much higher.

If you have sufficiently high code coverage I'd say that a successful unit test suite is enough confidence to check in code and let continuous integration take care of the rest.

Keep in mind that continuous integration and other types of automated builds are feedback mechanisms. If you institute rules where people are ‘punished' for checking in code that breaks the build, you effectively disable the automated build as a source of feedback.

Thus, the rest of these feedback mechanisms should be used rarely (if at all) on developer work stations.

Integration testing #

Integration testing comes in several sub-categories. You can integrate multiple classes from within the same library, or integrate multiple libraries while still using Test Doubles instead of out-of-process resources. At this level, the cost/confidence ratio is comparable to unit testing.

Subcutaneous testing #

A more full level of integration testing comes when all resources are integrated, but the tests are still driven by code instead of through the UI. While this gives a degree of confidence that is close to what you can get from a full systems test, the cost of setting up the entire testing environment is much higher. In many cases I consider this the realm of a dedicated testing department, as it's often a full-time job to maintain the suite of automation tools that enable such tests - particularly if we are talking about distributed systems.

System testing #

This is a test of the full system, often driven through the UI and supplemented with manual testing (such as exploratory testing). This provides the highest degree of confidence, but comes with a very high cost. It's often at this level we find formal acceptance tests.

The time before we can get feedback at this level is often considerable. It may take days or even weeks or months.

Understand the cost/confidence ratio #

The purpose of this post has been to talk about different ways we can get feedback when developing software. If we could get instantaneous feedback with guaranteed confidence it would be easy to choose, but as this is not the case we must understand the benefits and drawbacks of each mechanism.

It's important to realize that there are many mechanisms, and that we can combine them in a staggered approach where we start with fast feedback (like the compiler) with a low degree of confidence and then move through successive stages that each provide a higher level of confidence.

Comments

• ← Previous
• Archive
• Next →