From time to time I encounter people who attempt to express an API in terms of IQueryable<T>. That's almost always a bad idea. In this post, I'll explain why.

In short, the IQueryable<T> interface is one of the best examples of a Header Interface that .NET has to offer. It's almost impossible to fully implement it.

Please note that this post is about the problematic aspects of designing an API around the IQueryable<T> interface. It's not an attack on the interface itself, which has its place in the BCL. It's also not an attack on all the wonderful LINQ methods available on IEnumerable<T>.

You can say that IQueryable<T> is one big Liskov Substitution Principle (LSP)violation just waiting to happen. In the next two section, I will apply Postel's law to explain why that is.

Consuming IQueryable<T>

The first part of Postel's law applied to API design states that an API should be liberal in what it accepts. In other words, we are talking about input, so an API that consumes IQueryable<T> would take this generalized shape:

            IFoo SomeMethod(IQueryable<Bar>
q);

Is that a liberal requirement? It must certainly is not. Such an interface demands of any caller that they must be able to supply an implementation of IQueryable<Bar>. According to the LSP we must be able to supply any implementation without changing the correctness of the program. That goes for both the implementer of IQueryable<Bar> as well as the implementation of SomeMethod.

At this point it's important to keep in mind the purpose of IQueryable<T>: it's intended for implementation by query providers. In other words, this isn't just some sequence of Bar instances which can be filtered and projected; no, this is a query expression which is intended to be translated into a query somewhere else – most often some dialect of SQL.

That's quite a demand to put on the caller.

It's certainly a powerful interface (or so it would seem), but is it really necessary? Does SomeMethod really need to be able to perform arbitrarily complex queries against a data source?

In one recent discussion, it turns out that all the developer really wanted to do was to be able to select based on a handful of simple criteria. In another case, the developer only wanted to do simple paging.

Such requirements could be modeled much simpler without making huge demands on the caller. In both cases, we could provide specialized Query Objects instead, or perhaps even simpler just a set of specialized queries:

            IFoo FindById(int fooId);

            IFoo FindByCorrelationId(int correlationId);

Or, in the case of paging:

            IEnumerable<IFoo>
GetFoos(int page);

This is certainly much more liberal in that it requires the caller to supply only the required information in order to implement the methods. Designing APIs in terms of Role Interfaces instead of Header Interfaces makes the APIs much more flexible. This will enable you to respond to change.

Exposing IQueryable<T>

The other part of Postel's law states that an API should be conservative in what it sends. In other words, a method must guarantee that the data it returns conforms rigorously to the contract between caller and implementer.

A method returning IQueryable<T> would take this generalized shape:

            IQueryable<Bar>
GetBars();

When designing APIs, a huge part of the contract is defined by the interface (or base class). Thus, the return type of a method specifies a conservative guarantee about the returned data. In the case of returning IQueryable<Bar> the method thus guarantees that it will return a complete implementation of IQueryable<Bar>.

Is that conservative?

Once again invoking the LSP, a consumer must be able to do anything allowed by IQueryable<Bar> without changing the correctness of the program.

That's a big honking promise to make.

Who can keep that promise?

Current Implementations

Implementing IQueryable<T> is a huge undertaking. If you don't believe me, just take a look at the official Building an IQueryable provider series of blog posts. Even so, the interface is so flexible and expressive that with a single exception, it's always possible to write a query that a given provider can't translate.

Have you ever worked with LINQ to Entities or another ORM and received a NotSupportedException? Lots of people have. In fact, with a single exception, it's my firm belief that all existing implementations violate the LSP (in fact, I challenge my readers to refer me to a real, publicly available implementation of IQueryable<T> that can accept any expression I throw at it, and I'll ship a free copy of my book to the first reader to do so).

Furthermore, the subset of features that each implementation supports varies from query provider to query provider. An expression that can be translated by the Entity framework may not work with Microsoft's OData query provider.

The only implementation that fully implements IQueryable<T> is the in-memory implementation (and referring to this one does not earn you a free book). Ironically, this implementation can be considered a Null Object implementation and goes against the whole purpose of the IQueryable<T> interface exactly because it doesn't translate the expression to another language.

Why This Matters

You may think this is all a theoretical exercise, but it actually does matter. When writing Clean Code, it's important to design an API in such a way that it's clear what it does.

An interface like this makes false guarantees:

            public
            interface
            IRepository
          

{

    IQueryable<T>
Query<T>();

}

According to the LSP and Postel's law, it would seem to guarantee that you can write any query expression (no matter how complex) against the returned instance, and it would always work.

In practice, this is never going to happen.

Programmers who define such interfaces invariably have a specific ORM in mind, and they implicitly tend to stay within the bounds they know are safe for that specific ORM. This is a leaky abstraction.

If you have a specific ORM in mind, then be explicit about it. Don't hide it behind an interface. It creates the illusion that you can replace one implementation with another. In practice, that's impossible. Imagine attempting to provide an implementation over an Event Store.

The cake is a lie.

Abstract Factory is a tremendously useful pattern when used with Dependency Injection (DI). While I’ve repeatedly described how it can be used to solve various problems in DI, apparently I’ve never described how to implement one. As a comment to an older blog post of mine, Thomas Jaskula asks how I’d implement the IOrderShipperFactory.

To stay consistent with the old order shipper scenario, this blog post outlines three alternative ways to implement the IOrderShipperFactory interface.

To make it a bit more challenging, the implementation should create instances of the OrderShipper2 class, which itself has a dependency:

            
              
                public
              
            
            
              
                class
              
              
                OrderShipper2
               : 
            
              IOrderShipper
            
          

            {
          

                privatereadonlyIChannel channel;
          

                public OrderShipper2(IChannel channel)
          

               
{
          

                    if (channel
== null)
          

                        thrownewArgumentNullException("channel");
          

                    this.channel
= channel;
          

               
}
          

                publicvoid Ship(Order order)
          

               
{
          

                    
            
              //
Ship the order and
            
          

                    
            
              //
raise a domain event over this.channel
            
          

               
}
          

            }
          

In order to be able to create an instance of OrderShipper2, any factory implementation must be able to supply an IChannel instance.

Manually Coded Factory

The first option is to manually wire up the OrderShipper2 instance within the factory:

            
              
                public
              
            
            
              
                class
              
              
                ManualOrderShipperFactory
               :
          

                
            
              IOrderShipperFactory
            
          

            {
          

                privatereadonlyIChannel channel;
          

                public ManualOrderShipperFactory(IChannel channel)
          

               
{
          

                    if (channel
== null)
          

                        thrownewArgumentNullException("channel");
          

                    this.channel
= channel;
          

               
}
          

                publicIOrderShipper Create()
          

               
{
          

                    returnnewOrderShipper2(this.channel);
          

               
}
          

            }
          

This has the advantage that it’s easy to understand. It can be unit tested and implemented in the same library that also contains OrderShipper2 itself. This means that any client of that library is supplied with a read-to-use implementation.

The disadvantage of this approach is that if/when the constructor of OrderShipper2 changes, the ManualOrderShipperFactory class must also be corrected. Pragmatically, this may not be a big deal, but one could successfully argue that this violates the Open/Closed Principle.

Container-based Factory

Another option is to make the implementation a thin Adapter over a DI Container – in this example Castle Windsor:

            
              
                public
              
            
            
              
                class
              
              
                ContainerFactory
               : 
            
              IOrderShipperFactory
            
          

            {
          

                privateIWindsorContainer container;
          

                public ContainerFactory(IWindsorContainer container)
          

               
{
          

                    if (container
== null)
          

                        thrownewArgumentNullException("container");
          

                    this.container
= container;
          

               
}
          

                publicIOrderShipper Create()
          

               
{
          

                    returnthis.container.Resolve<IOrderShipper>();
          

               
}
          

            }
          

But wait! Isn’t this an application of the Service Locator anti-pattern? Not if this class is part of the Composition Root.

If this implementation was placed in the same library as OrderShipper2 itself, it would mean that the library would have a hard dependency on the container. In such a case, it would certainly be a Service Locator.

However, when a Composition Root already references a container, it makes sense to place the ContainerFactory class there. This changes its role to the pure infrastructure component it really ought to be. This seems more SOLID, but the disadvantage is that there’s no longer a ready-to-use implementation packaged together with the LazyOrderShipper2 class. All new clients must supply their own implementation.

Dynamic Proxy

The third option is to basically reduce the principle behind the container-based factory to its core. Why bother writing even a thin Adapter if one can be automatically generated.

With Castle Windsor, the Typed Factory Facility makes this possible:

            container.AddFacility<TypedFactoryFacility>();
          

            container.Register(
            
              Component
            
          

               
.For<IOrderShipperFactory>()
          

               
.AsFactory());
          

            
              
                var
              
            
             factory
=
          

               
container.Resolve<IOrderShipperFactory>();
          

There is no longer any code which implements IOrderShipperFactory. Instead, a class conceptually similar to the ContainerFactory class above is dynamically generated and emitted at runtime.

While the code never materializes, conceptually, such a dynamically emitted implementation is still part of the Composition Root.

This approach has the advantage that it’s very DRY, but the disadvantages are similar to the container-based implementation above: there’s no longer a ready-to-use implementation. There’s also the additional disadvantage that out of the three alternative here outlined, the proxy-based implementation is the most difficult to understand.

For years, layered application architecture has been a de-facto standard for loosely coupled architectures, but the question is: does the layering really provide benefit?

In theory, layering is a way to decouple concerns so that UI concerns or data access technologies don’t pollute the domain model. However, this style of architecture seems to come at a pretty steep price: there’s a lot of mapping going on between the layers. Is it really worth the price, or is it OK to define data structures that cut across the various layers?

The short answer is that if you cut across the layers, it’s no longer a layered application. However, the price of layering may still be too high.

In this post I’ll examine the problem and the proposed solution and demonstrate why none of them are particularly good. In the end, I’ll provide a pointer going forward.

Proper Layering

To understand the problem with layering, I’ll describe a fairly simple example. Assume that you are building a music service and you’ve been asked to render a top 10 list for the day. It would have to look something like this in a web page:

As part of rending the list, you must color the Movement values accordingly using CSS.

A properly layered architecture would look something like this:

Each layer defines some services and some data-carrying classes (Entities, if you want to stick with the Domain-Driven Design terminology). The Track class is defined by the Domain layer, while the TopTrackViewModel class is defined in the User Interface layer, and so on. If you are wondering about why Track is used to communicate both up and down, this is because the Domain layer should be the most decoupled layer, so the other layers exist to serve it. In fact, this is just a vertical representation of the Ports and Adapters architecture, with the Domain Model sitting in the center.

This architecture is very decoupled, but comes at the cost of much mapping and seemingly redundant repetition. To demonstrate why that is, I’m going to show you some of the code. This is an ASP.NET MVC application, so the Controller is an obvious place to start:

            
              
                public
              
            
            
              
                ViewResult
               Index()
          

            {
          

                var date
= DateTime.Now.Date;
          

                var topTracks
= this.trackService.GetTopFor(date);
          

                returnthis.View(topTracks);
          

            }
          

This doesn’t look so bad. It asks an ITrackService for the top tracks for the day and returns a list of TopTrackViewModel instances. This is the implementation of the track service:

            
              
                public
              
            
            
              
                IEnumerable
              <TopTrackViewModel>
GetTopFor(
          

                DateTime date)
          

            {
          

                var today
= DateTime.Now.Date;
          

                var yesterDay
= today - TimeSpan.FromDays(1);
          

                var todaysTracks
= 
          

                    this.repository.GetTopFor(today).ToList();
          

                var yesterdaysTracks
= 
          

                    this.repository.GetTopFor(yesterDay).ToList();
          

                var length
= todaysTracks.Count;
          

                var positions
= Enumerable.Range(1, length);
          

                returnfrom tt in todaysTracks.Zip(
          

                           
positions, (t, p) =>
          

                                new {
Position = p, Track = t })
          

                        let yp
= (from yt in yesterdaysTracks.Zip(
          

                                       
positions, (t, p) => 
          

                                            
            
              new
            
          

                                           
{
          

                                               
Position = p,
          

                                               
Track = t
          

                                           
})
          

                                    where yt.Track.Id
== tt.Track.Id
          

                                    select yt.Position)
          

                                   
.DefaultIfEmpty(-1)
          

                                   
.Single()
          

                        let cssClass
= GetCssClass(tt.Position, yp)
          

                        selectnew
            
              TopTrackViewModel
            
          

                       
{
          

                           
Position = tt.Position,
          

                           
Name = tt.Track.Name,
          

                           
Artist = tt.Track.Artist,
          

                           
CssClass = cssClass
          

                       
};
          

            }
          

            
              
                private
              
            
            
              
                static
              
              
                string
               GetCssClass(
          

                int todaysPosition, int yesterdaysPosition)
          

            {
          

                if (yesterdaysPosition
< 0)
          

                    return"new";
          

                if (todaysPosition
< yesterdaysPosition)
          

                    return"up";
          

                if (todaysPosition
== yesterdaysPosition)
          

                    return"same";
          

                return"down";
          

            }
          

While that looks fairly complex, there’s really not a lot of mapping going on. Most of the work is spent getting the top 10 track for today and yesterday. For each position on today’s top 10, the query finds the position of the same track on yesterday’s top 10 and creates a TopTrackViewModel instance accordingly.

Here’s the only mapping code involved:

            
              
                select
              
            
            
              
                new
              
            
            
              TopTrackViewModel
            
          

            {
          

               
Position = tt.Position,
          

               
Name = tt.Track.Name,
          

               
Artist = tt.Track.Artist,
          

               
CssClass = cssClass
          

            };
          

This maps from a Track (a Domain class) to a TopTrackViewModel (a UI class).

This is the relevant implementation of the repository:

            
              
                public
              
            
            
              
                IEnumerable
              <Track>
GetTopFor(DateTime date)
          

            {
          

                var dbTracks
= this.GetTopTracks(date);
          

                foreach (var dbTrack in dbTracks)
          

               
{
          

                    yieldreturnnewTrack(
          

                       
dbTrack.Id,
          

                       
dbTrack.Name,
          

                       
dbTrack.Artist);
          

               
}
          

            }
          

You may be wondering about the translation from DbTrack to Track. In this case you can assume that the DbTrack class is a class representation of a database table, modeled along the lines of your favorite ORM. The Track class, on the other hand, is a proper object-oriented class which protects its invariants:

            
              
                public
              
            
            
              
                class
              
            
            
              Track
            
          

            {
          

                privatereadonlyint id;
          

                privatestring name;
          

                privatestring artist;
          

                public Track(int id, string name, string artist)
          

               
{
          

                    if (name
== null)
          

                        thrownewArgumentNullException("name");
          

                    if (artist
== null)
          

                        thrownewArgumentNullException("artist");
          

                    this.id
= id;
          

                    this.name
= name;
          

                    this.artist
= artist;
          

               
}
          

                publicint Id
          

               
{
          

                    get { returnthis.id;
}
          

               
}
          

                publicstring Name
          

               
{
          

                    get { returnthis.name;
}
          

                    
            
              set
            
          

                   
{
          

                        if (value == null)
          

                            thrownewArgumentNullException("value");
          

                        this.name
= value;
          

                   
}
          

               
}
          

                publicstring Artist
          

               
{
          

                    get { returnthis.artist;
}
          

                    
            
              set
            
          

                   
{
          

                        if (value == null)
          

                            thrownewArgumentNullException("value");
          

                        this.artist
= value;
          

                   
}
          

               
}
          

            }
          

No ORM I’ve encountered so far has been able to properly address such invariants – particularly the non-default constructor seems to be a showstopper. This is the reason a separate DbTrack class is required, even for ORMs with so-called POCO support.

In summary, that’s a lot of mapping. What would be involved if a new field is required in the top 10 table? Imagine that you are being asked to provide the release label as an extra column.

1. A Label column must be added to the database schema and the DbTrack class.
2. A Label property must be added to the Track class.
3. The mapping from DbTrack to Track must be updated.
4. A Label property must be added to the TopTrackViewModel class.
5. The mapping from Track to TopTrackViewModel must be updated.
6. The UI must be updated.

That’s a lot of work in order to add a single data element, and this is even a read-only scenario! Is it really worth it?

Cross-Cutting Entities

Is strict separation between layers really so important? What would happen if Entities were allowed to travel across all layers? Would that really be so bad?

Such an architecture is often drawn like this:

Now, a single Track class is allowed to travel from layer to layer in order to avoid mapping. The controller code hasn’t really changed, although the model returned to the View is no longer a sequence of TopTrackViewModel, but simply a sequence of Track instances:

            
              
                public
              
            
            
              
                ViewResult
               Index()
          

            {
          

                var date
= DateTime.Now.Date;
          

                var topTracks
= this.trackService.GetTopFor(date);
          

                returnthis.View(topTracks);
          

            }
          

The GetTopFor method also looks familiar:

            
              
                public
              
            
            
              
                IEnumerable
              <Track>
GetTopFor(DateTime date)
          

            {
          

                var today
= DateTime.Now.Date;
          

                var yesterDay
= today - TimeSpan.FromDays(1);
          

                var todaysTracks
= 
          

                    this.repository.GetTopFor(today).ToList();
          

                var yesterdaysTracks
= 
          

                    this.repository.GetTopFor(yesterDay).ToList();
          

                var length
= todaysTracks.Count;
          

                var positions
= Enumerable.Range(1, length);
          

                returnfrom tt in todaysTracks.Zip(
          

                           
positions, (t, p) => 
          

                                new {
Position = p, Track = t })
          

                        let yp
= (from yt in yesterdaysTracks.Zip(
          

                                       
positions, (t, p) =>
          

                                            
            
              new
            
          

                                           
{
          

                                               
Position = p,
          

                                               
Track = t
          

                                           
})
          

                                    where yt.Track.Id
== tt.Track.Id
          

                                    select yt.Position)
          

                                   
.DefaultIfEmpty(-1)
          

                                   
.Single()
          

                        let cssClass
= GetCssClass(tt.Position, yp)
          

                        select Enrich(
          

                           
tt.Track, tt.Position, cssClass);
          

            }
          

            
              
                private
              
            
            
              
                static
              
              
                string
               GetCssClass(
          

                int todaysPosition, int yesterdaysPosition)
          

            {
          

                if (yesterdaysPosition
< 0)
          

                    return"new";
          

                if (todaysPosition
< yesterdaysPosition)
          

                    return"up";
          

                if (todaysPosition
== yesterdaysPosition)
          

                    return"same";
          

                return"down";
          

            }
          

            
              
                private
              
            
            
              
                static
              
              
                Track
               Enrich(
          

                Track track, int position, string cssClass)
          

            {
          

               
track.Position = position;
          

               
track.CssClass = cssClass;
          

                return track;
          

            }
          

Whether or not much has been gained is likely to be a subjective assessment. While mapping is no longer taking place, it’s still necessary to assign a CSS Class and Position to the track before handing it off to the View. This is the responsibility of the new Enrich method:

            
              
                private
              
            
            
              
                static
              
              
                Track
               Enrich(
          

                Track track, int position, string cssClass)
          

            {
          

               
track.Position = position;
          

               
track.CssClass = cssClass;
          

                return track;
          

            }
          

If not much is gained at the UI layer, perhaps the data access layer has become simpler? This is, indeed, the case:

            
              
                public
              
            
            
              
                IEnumerable
              <Track>
GetTopFor(DateTime date)
          

            {
          

                returnthis.GetTopTracks(date);
          

            }
          

If, hypothetically, you were asked to add a label to the top 10 table it would be much simpler:

1. A Label column must be added to the database schema and the Track class.
2. The UI must be updated.

This looks good. Are there any disadvantages? Yes, certainly. Consider the Track class:

            
              
                public
              
            
            
              
                class
              
            
            
              Track
            
          

            {
          

                publicint Id
{ get; set;
}
          

                publicstring Name
{ get; set;
}
          

                publicstring Artist
{ get; set;
}
          

                publicint Position
{ get; set;
}
          

                publicstring CssClass
{ get; set;
}
          

            }
          

It also looks simpler than before, but this is actually not particularly beneficial, as it doesn’t protect its invariants. In order to play nice with the ORM of your choice, it must have a default constructor. It also has automatic properties. However, most insidiously, it also somehow gained the Position and CssClass properties.

What does the Position property imply outside of the context of a top 10 list? A position in relation to what?

Even worse, why do we have a property called CssClass? CSS is a very web-specific technology so why is this property available to the Data Access and Domain layers? How does this fit if you are ever asked to build a native desktop app based on the Domain Model? Or a REST API? Or a batch job?

When Entities are allowed to travel along layers, the layers basically collapse. UI concerns and data access concerns will inevitably be mixed up. You may think you have layers, but you don’t.

Is that such a bad thing, though?

Perhaps not, but I think it’s worth pointing out:

The choice is whether or not you want to build a layered application. If you want layering, the separation must be strict. If it isn’t, it’s not a layered application.

There may be great benefits to be gained from allowing Entities to travel from database to user interface. Much mapping cost goes away, but you must realize that then you’re no longer building a layered application – now you’re building a web application (or whichever other type of app you’re initially building).

Further Thoughts

It’s a common question: how hard is it to add a new field to the user interface?

The underlying assumption is that the field must somehow originate from a corresponding database column. If this is the case, mapping seems to be in the way.

However, if this is the major concern about the application you’re currently building, it’s a strong indication that you are building a CRUD application. If that’s the case, you probably don’t need a Domain Model at all. Go ahead and let your ORM POCO classes travel up and down the stack, but don’t bother creating layers: you’ll be building a monolithic application no matter how hard you try not to.

In the end it looks as though none of the options outlined in this article are particularly good. Strict layering leads to too much mapping, and no mapping leads to monolithic applications. Personally, I’ve certainly written quite a lot of strictly layered applications, and while the separation of concerns was good, I was never happy with the mapping overhead.

At the heart of the problem is the CRUDy nature of many applications. In such applications, complex object graphs are traveling up and down the stack. The trick is to avoid complex object graphs.

Move less data around and things are likely to become simpler. This is one of the many interesting promises of CQRS, and particularly it’s asymmetric approach to data.

To be honest, I wish I had fully realized this when I started writing my book, but when I finally realized what I’d implicitly felt for long, it was to late to change direction. Rest assured that nothing in the book is fundamentally flawed. The patterns etc. still hold, but the examples could have been cleaner if the sample applications had taken a CQRS-like direction instead of strict layering.

A common criticism of loosely coupled code is that it’s harder to understand. How do you see the big picture of an application when loose coupling is everywhere? When the entire code base has been programmed against interfaces instead of concrete classes, how do we understand how the objects are wired and how they interact?

In this post, I’ll provide answers on various levels, from high-level architecture over object-oriented principles to more nitty-gritty code. Before I do that, however, I’d like to pose a set of questions you should always be prepared to answer.

Mu

My first reaction to that sort of question is: you say loosely coupled code is harder to understand. Harder than what?

If we are talking about a non-trivial application, odds are that it’s going to take some time to understand the code base – whether or not it’s loosely coupled. Agreed: understanding a loosely coupled code base takes some work, but so does understanding a tightly coupled code base. The question is whether it’s harder to understand a loosely coupled code base?

Imagine that I’m having a discussion about this subject with Mary Rowan from my book.

Mary: “Loosely coupled code is harder to understand.”

Me: “Why do you think that is?”

Mary: “It’s very hard to navigate the code base because I always end up at an interface.”

Me: “Why is that a problem?”

Mary: “Because I don’t know what the interface does.”

At this point I’m very tempted to answer Mu. An interfaces doesn’t do anything – that’s the whole point of it. According to the Liskov Substitution Principle (LSP), a consumer shouldn’t have to care about what happens on the other side of the interface.

However, developers used to tightly coupled code aren’t used to think about services in this way. They are used to navigate the code base from consumer to service to understand how the two of them interact, and I will gladly admit this: in principle, that’s impossible to do in a loosely coupled code base. I’ll return to this subject in a little while, but first I want to discuss some strategies for understanding a loosely coupled code base.

Architecture and Documentation

Yes: documentation. Don’t dismiss it. While I agree with Uncle Bob and like-minded spirits that the code is the documentation, a two-page document that outlines the Big Picture might save you from many hours of code archeology.

The typical agile mindset is to minimize documentation because it tends to lose touch with the code base, but even so, it should be possible to maintain a two-page high-level document so that it stays up to date. Consider the alternative: if you have so much architectural churn that even a two-page overview regularly falls behind, then you’re probably having a greater problem than understanding your loosely coupled code base.

Maintaining such a document isn’t adverse to the agile spirit. You’ll find the same advice in Lean Architecture (p. 127). Don’t underestimate the value of such a document.

See the Forest Instead of the Trees

Understanding a loosely coupled code base typically tends to require a shift of mindset.

Recall my discussion with Mary Rowan. The criticism of loose coupling is that it’s difficult to understand which collaborators are being invoked. A developer like Mary Rowan is used to learn a code base by understanding all the myriad concrete details of it. In effect, while there may be ‘classes’ around, there are no abstractions in place. In order to understand the behavior of a user interface element, it’s necessary to also understand what’s happening in the database – and everything in between.

A loosely coupled code base shouldn’t be like that.

The entire purpose of loose coupling is that we should be able to reason about a part (a ‘unit’, if you will) without understanding the whole.

In a tightly coupled code base, it’s often impossible to see the forest for the trees. Although we developers are good at relating to details, a tightly coupled code base requires us to be able to contain the entire code base in our heads in order to understand it. As the size of the code base grows, this becomes increasingly difficult.

In a loosely coupled code base, on the other hand, it should be possible to understand smaller parts in isolation. However, I purposely wrote “should be”, because that’s not always the case. Often, a so-called “loosely coupled” code base violates basic principles of object-oriented design.

RAP

The criticism that it’s hard to see “what’s on the other side of an interface” is, in my opinion, central. It betrays a mindset which is still tightly coupled.

In many code bases there’s often a single implementation of a given interface, so developers can be forgiven if they think about an interface as only a piece of friction that prevents them from reaching the concrete class on the other side. However, if that’s the case with most of the interfaces in a code base, it signifies a violation of the Reused Abstractions Principle (RAP) more than it signifies loose coupling.

Jim Cooper, a reader of my book, put it quite eloquently on the book’s forum:

“So many people think that using an interface magically decouples their code. It doesn't. It only changes the nature of the coupling. If you don't believe that, try changing a method signature in an interface - none of the code containing method references or any of the implementing classes will compile. I call that coupled”

Refactoring tools aside, I completely agree with this statement. The RAP is a test we can use to verify whether or not an interface is truly reusable – what better test is there than to actually reuse your interfaces?

The corollary of this discussion is that if a code base is massively violating the RAP then it’s going to be hard to understand. It has all the disadvantages of loose coupling with few of the benefits. If that’s the case, you would gain more benefit from making it either more tightly coupled or truly loosely coupled.

What does “truly loosely coupled” mean?

LSP

According to the LSP a consumer must not concern itself with “what’s on the other side of the interface”. It should be possible to replace any implementation with any other implementation of the same interface without changing the correctness of the program.

This is why I previously said that in a truly loosely coupled code base, it isn’t ‘hard’ to understand “what’s on the other side of the interface” – it’s impossible. At design-time, there’s nothing ‘behind’ the interface. The interface is what you are programming against. It’s all there is.

Mary has been listening to all of this, and now she protests:

Mary: “At run-time, there’s going to be a concrete class behind the interface.”

Me (being annoyingly pedantic): “Not quite. There’s going to be an instance of a concrete class which the consumer invokes via the interface it implements.”

Mary: “Yes, but I still need to know which concrete class is going to be invoked.”

Me: “Why?”

Mary: “Because otherwise I don’t know what’s going to happen when I invoke the method.”

This type of answer often betrays a much more fundamental problem in a code base.

CQS

Now we are getting into the nitty-gritty details of class design. What would you expect that the following method does?

              
                
                  public
                
              
              
                
                  List
                <Order>
GetOpenOrders(Customer customer)
            

The method name indicates that it gets open orders, and the signature seems to back it up. A single database query might be involved, since this looks a lot like a read-operation. A quick glance at the implementation seems to verify that first impression:

            
              
                public
              
            
            
              
                List
              <Order>
GetOpenOrders(Customer customer)
          

            {
          

                var orders
= GetOrders(customer);
          

                return (from o in orders
          

                        where o.Status
== OrderStatus.Open
          

                        select o).ToList();
          

            }
          

Is it safe to assume that this is a side-effect-free method call? As it turns out, this is far from the case in this particular code base:

            
              
                private
              
            
            
              
                List
              <Order>
GetOrders(Customer customer)
          

            {
          

                var gw
= newCustomerGateway(this.ConnectionString);
          

                var orders
= gw.GetOrders(customer);
          

               
AuditOrders(orders);
          

               
FixCustomer(gw, orders, customer);
          

                return orders;
          

            }
          

The devil is in the details. What does AuditOrders do? And what does FixCustomer do? One method at a time:

            
              
                private
              
            
            
              
                void
               AuditOrders(List<Order>
orders)
          

            {
          

                var user
= Thread.CurrentPrincipal.Identity.ToString();
          

                var gw
= newOrderGateway(this.ConnectionString);
          

                foreach (var o in orders)
          

               
{
          

                    var clone
= o.Clone();
          

                    var ar
= new
            
              AuditRecord
            
          

                   
{
          

                       
Time = DateTime.Now,
          

                       
User = user
          

                   
};
          

                   
clone.AuditTrail.Add(ar);
          

                   
gw.Update(clone);
          

                    
            
              //
We don't want the consumer to see the audit trail.
            
          

                   
o.AuditTrail.Clear();
          

               
}
          

            }
          

OK, it turns out that this method actually makes a copy of each and every order and updates that copy, writing it back to the database in order to leave behind an audit trail. It also mutates each order before returning to the caller. Not only does this method result in an unexpected N+1 problem, it also mutates its input, and perhaps even more surprising, it’s leaving the system in a state where the in-memory object is different from the database. This could lead to some pretty interesting bugs.

Then what happens in the FixCustomer method? This:

            
              //
Fixes the customer status field if there were orders
            
          

            
              //
added directly through the database.
            
          

            
              
                private
              
            
            
              
                static
              
              
                void
               FixCustomer(CustomerGateway gw,
          

                List<Order>
orders, Customer customer)
          

            {
          

                var total
= orders.Sum(o => o.Total);
          

                if (customer.Status
!= CustomerStatus.Preferred
          

                   
&& total > PreferredThreshold)
          

               
{
          

                   
customer.Status = CustomerStatus.Preferred;
          

                   
gw.Update(customer);
          

               
}
          

            }
          

Another potential database write operation, as it turns out – complete with an apology. Now that we’ve learned all about the details of the code, even the GetOpenOrders method is beginning to look suspect. The GetOrders method returns all orders, with the side effect that all orders were audited as having been read by the user, but the GetOpenOrders filters the output. In the end, it turns out that we can’t even trust the audit trail.

While I must apologize for this contrived example of a Transaction Script, it’s clear that when code looks like that, it’s no wonder why developers think that it’s necessary to contain the entire code base in their heads. When this is the case, interfaces are only in the way.

However, this is not the fault of loose coupling, but rather a failure to adhere to the very fundamental principle of Command-Query Separation (CQS). You should be able to tell from the method signature alone whether invoking the method will or will not have side-effects. This is one of the key messages from Clean Code: the method name and signature is an abstraction. You should be able to reason about the behavior of the method from its declaration. You shouldn’t have to read the code to get an idea about what it does.

Abstractions always hide details. Method declarations do too. The point is that you should be able to read just the method declaration in order to gain a basic understanding of what’s going on. You can always return to the method’s code later in order to understand detail, but reading the method declaration alone should provide the Big Picture.

Strictly adhering to CQS goes a long way in enabling you to understand a loosely coupled code base. If you can reason about methods at a high level, you don’t need to see “the other side of the interface” in order to understand the Big Picture.

Stack Traces

Still, even in a loosely coupled code base with great test coverage, integration issues arise. While each class works fine in isolation, when you integrate them, sometimes the interactions between them cause errors. This is often because of incorrect assumptions about the collaborators, which often indicates that the LSP was somehow violated.

To understand why such errors occur, we need to understand which concrete classes are interacting. How do we do that in a loosely coupled code base?

That’s actually easy: look at the stack trace from your error report. If your error report doesn’t include a stack trace, make sure that it’s going to do that in the future.

The stack trace is one of the most important troubleshooting tools in a loosely coupled code base, because it’s going to tell you exactly which classes were interacting when an exception was thrown.

Furthermore, if the code base also adheres to the Single Responsibility Principle and the ideals from Clean Code, each method should be very small (under 15 lines of code). If that’s the case, you can often understand the exact nature of the error from the stack trace and the error message alone. It shouldn’t even be necessary to attach a debugger to understand the bug, but in a pinch, you can still do that.

Tooling

Returning to the original question, I often hear people advocating tools such as IDE add-ins which support navigation across interfaces. Such tools might provide a menu option which enables you to “go to implementation”. At this point it should be clear that such a tool is mainly going to be helpful in code bases that violate the RAP.

(I’m not naming any particular tools here because I’m not interested in turning this post into a discussion about the merits of various specific tools.)

Conclusion

It’s the responsibility of the loosely coupled code base to make sure that it’s easy to understand the Big Picture and that it’s easy to work with. In the end, that responsibility falls on the developers who write the code – not the developer who’s trying to understand it.

Recently I received a question from Kelly Sommers about good ways to refactor away from Factory Overload. Basically, she’s working in a code base where there’s an explosion of Abstract Factories which seems to be counter-productive. In this post I’ll take a look at the example problem and propose a set of alternatives.

An Abstract Factory (and its close relative Product Trader) can serve as a solution to various challenges that come up when writing loosely coupled code (chapter 6 of my book describes the most common scenarios). However, introducing an Abstract Factory may be a leaky abstraction, so don’t do it blindly. For example, an Abstract Factory is rarely the best approach to address lifetime management concerns. In other words, the Abstract Factory has to make sense as a pure model element.

That’s not the case in the following example.

Problem Statement

The question centers around a code base that integrates with a database closely guarded by DBA police. Apparently, every single database access must happen through a set of very fine-grained stored procedures.

For example, to update the first name of a user, a set of stored procedures exist to support this scenario, depending on the context of the current application user:

As this table demonstrates, not only is there a stored procedure for each user context, but the parameter name differs as well. However, in this particular case it seems as though there’s a pattern to the names.

If this pattern is consistent, I think the easiest way to address these variations would be to algorithmically build the strings from a couple of templates.

However, this is not the route taken by Kelly’s team, so I assume that things are more complicated than that; apparently, a templated approach is not viable, so for the rest of  this article I’m going to assume that it’s necessary to write at least some code to address each case individually.

The current solution that Kelly’s team has implemented is to use an Abstract Factory (Product Trader) to translate the user type into an appropriate IUserFirstNameModifier instance. From the consuming code, it looks like this:

            
              
                var
              
            
             modifier
= factory.Create(UserTypes.Admin);
          

            modifier.Commit("first");
          

where the factory variable is an instance of the IUserFirstNameModifierFactory interface. This is certainly loosely coupled, but looks like a leaky abstraction. Why is a factory needed? It seems that its single responsibility is to translate a UserTypes instance (an enum) into an IUserFirstNameModifier. There’s a code smell buried here – try to spot it before you read on :)

Proposed Solution

Kelly herself suggests an alternative involving a concrete Builder which can create instances of a single concrete UserFirstNameModifier with or without an implicit conversion:

            
              //
Implicit conversion.
            
          

            
              
                UserFirstNameModifier
              
            
             modifier1
= 
          

               
builder.WithUserType(UserTypes.Guest);
          

            
              //
Without implicit conversion.
            
          

            
              
                var
              
            
             modifier2
= builder
          

               
.WithUserType(UserTypes.Restricted)
          

               
.Create();
          

While this may seem to reduce the number of classes involved, it has several drawbacks:

• First of all, the Fluent Builder pattern implies that you can forgo invoking any of the WithXyz methods (WithUserType) and just accept all the default values encapsulated in the builder. This again implies that there’s a default user type, which may or may not make sense in that particular domain. Looking at Kelly’s code, UserTypes is an enum (and thus has a default value), so if WithUserType isn’t invoked, the Create method defaults to UserTypes.Admin. That’s a bit too implicit for my taste.
• Since all involved classes are now concrete, the proposed solution isn’t extensibile (and by corollary hard to unit test).
• The builder is essentially a big switch statement.

Both the current implementation and the proposed solution involves passing an enum as a method parameter to a different class. If you’ve read and memorized Refactoring you should by now have recognized both a code smell and the remedy.

Alternative 1a: Make UserType Polymorphic

The code smell is Feature Envy and a possible refactoring is to replace the enum with a Strategy. In order to do that, an IUserType interface is introduced:

            
              
                public
              
            
            
              
                interface
              
            
            
              IUserType
            
          

            {
          

                IUserFirstNameModifer CreateUserFirstNameModifier();
          

            }
          

Usage becomes as simple as this:

            
              
                var
              
            
             modifier
= userType.CreateUserFirstNameModifier();
          

Obviously, more methods can be added to IUserType to support other update operations, but care should be taken to avoid creating a Header Interface.

While this solution is much more object-oriented, I’m still not quite happy with it, because apparently, the context is a CQRS style architecture. Since an update operation is essentially a Command, then why model the implementation along the lines of a Query? Both Abstract Factory and Factory Method patterns represent Queries, so it seems redundant in this case. It should be possible to apply the Hollywood Principle here.

Alternative 1b: Tell, Don’t Ask

Why have the user type return an modifier? Why can’t it perform the update itself? The IUserType interface should be changed to something like this:

            
              
                public
              
            
            
              
                interface
              
            
            
              IUserType
            
          

            {
          

                void CommitUserFirtName(string firstName);
          

            }
          

This makes it easier for the consumer to commit the user’s first name because it can be done directly on the IUserType instance instead of first creating the modifier.

It also makes it much easier to unit test the consumer because there’s no longer a mix of Command and Queries within the same method. From Growing Object-Oriented Software we know that Queries should be modeled with Stubs and Commands with Mocks, and if you’ve ever tried mixing the two you know that it’s a sort of interaction that should be minimized.

Alternative 2a: Distinguish by Type

While I personally like alternative 1b best, it may not be practical in all situations, so it’s always valuable to examine other alternatives.

The root cause of the problem is that there’s a lot of stored procedures. I want to reiterate that I still think that the absolutely easiest solution would be to generate a SqlCommand from a string template, but given that this article assumes that this isn’t possible or desirable, it follows that code must be written for each stored procedure.

Why not simply define an interface for each one? As an example, to update the user’s first name in the context of being an ‘Admin’ user, this Role Interface can be used:

            
              
                public
              
            
            
              
                interface
              
            
            
              IUserFirstNameAdminModifier
            
          

            {
          

                void Commit(string firstName);
          

            }
          

Similar interfaces can be defined for the other user types, such as IUserFirstNameRestrictedModifier, IUserFirstNameGuestModifier and so on.

This is a very simple solution; it’s easy to implement, but risks violating the Reused Abstractions Principle (RAP).

Alternative 2b: Distinguish by Generic Type

The problem with introducing interfaces like IUserFirstNameAdminModifier, IUserFirstNameRestrictedModifier, IUserFirstNameGuestModifier etc. is that they differ only by name. The Commit method is the same for all these interfaces, so this seems to violate the RAP. It’d be better to merge all these interfaces into a single interface, which is what Kelly’s team is currently doing. However, the problem with this is that the type carries no information about the role that the modifier is playing.

Another alternative is to turn the modifier interface into a generic interface like this:

            
              
                public
              
            
            
              
                interface
              
              
                IUserFirstNameModifier
              <T> 
          

                where T
: 
            
              IUserType
            
          

            {
          

                void Commit(string firstName);
          

            }
          

The IUserType is a Marker Interface, so .NET purists are not going to like this solution, since the .NET Type Design Guidelines recommend against using Marker Interfaces. However, it’s impossible to constrain a generic type argument against an attribute, so the party line solution is out of the question.

This solution ensures that consumers can now have dependencies on IUserFirstNameModifier<AdminUserType>, IUserFirstNameModifier<RestrictedUserType>, etc.

However, the need for a marker interface gives off another odeur.

Alternative 3: Distinguish by Role

The problem at the heart of alternative 2 is that it attempts to use the type of the interfaces as an indicator of the roles that Services play. It’s seems that making the type distinct works against the RAP, but when the RAP is applied, the type becomes ambiguous.

However, as Ted Newardpoints out in his excellent series on Multiparadigmatic .NET, the type is only one axis of variability among many. Perhaps, in this case, it may be much easier to use the name of the dependency to communicate its role instead of the type.

Given a single, ambiguous IUserFirstNameModifier interface (just as in the original problem statement), a consumer can distinguish between the various roles of modifiers by their names:

            
              
                public
              
            
            
              
                partial
              
              
                class
              
            
            
              SomeConsumer
            
          

            {
          

                privatereadonlyIUserFirstNameModifier adminModifier;
          

                privatereadonlyIUserFirstNameModifier guestModifier;
          

                public SomeConsumer(
          

                    IUserFirstNameModifier adminModifier,
          

                    IUserFirstNameModifier guestModifier)
          

               
{
          

                    this.adminModifier
= adminModifier;
          

                    this.guestModifier
= guestModifier;
          

               
}
          

                publicvoid DoSomething()
          

               
{
          

                    if (this.UseAdmin)
          

                        this.adminModifier.Commit("first");
          

                    
            
              else
            
          

                        this.guestModifier.Commit("first");
          

               
}
          

            }
          

Now it’s entirely up to the Composition Root to compose SomeConsumer with the correct modifiers, and while this can be done manually, it’s an excellent case for a DI Container and a bit of Convention over Configuration.

Conclusion

I’m sure that if I’d spent more time analyzing the problem I could have come up with more alternatives, but this post is becoming long enough already.

Of the alternatives I’ve suggested here, I prefer 1b or 3, depending on the exact requirements.

Asynchronous message passing combined with eventual consistency makes it possible to build very scalable systems. However, sometimes eventual consistency isn’t appropriate in parts of the system, while it’s acceptable in other parts. How can a consistent architecture be defined to fit both ACID and eventual consistency? This article provides an answer.

The case of an online game

Last week I visited Pixel Pandemic, a company that produces browser-based MMORPGs. Since each game world has lots of players who can all potentially interact with each other, scalability is very important.

In traditional line of business applications, eventual consistency is often an excellent fit because the application is a projection of the real world. My favorite example is an inventory system: it models what’s going on in one or more physical warehouses, but the real world is the ultimate source of truth. A warehouse worker might accidentally drop and damage some of the goods, in which case the application must adjust after the fact.

In other words, the information contained within line of business applications tend to lag after the real world. It’s impossible to guarantee that the application is always consistent with the real world, so eventual consistency is a natural fit.

That’s not the case with an online game world. The game world itself is the source of truth, and it must be internally consistent at all times. As an example, in Zombie Pandemic, players fight against zombies and may take damage along the way. Players can heal themselves, but they would prefer (I gather) that the healing action takes place immediately, and not some time in the future where the character might be dead. Similarly, when a player hits a zombie, they’d prefer to apply the damage immediately. (However, I think that even here, eventual consistency might provide some interesting game mechanics, but that’s another discussion.)

While discussing these matters with the nice people in Pixel Pandemic, it turned out that while some parts of the game world have to be internally consistent, it’s perfectly acceptable to use eventual consistency in other cases. One example is the game’s high score table. While a single player should have a consistent view of his or her own score, it’s acceptable if the high score table lags a bit.

At this point it seemed clear that this particular online game could use an appropriate combination of ACID and eventual consistency, and I think this conclusion can be generalized. The question now becomes: how can a consistent architecture encompass both types of consistency?

Problem statement

With the above example scenario in mind the problem statement can be generalized:

Given that an application should apply a mix of ACID and eventual consistency, how can a consistent architecture be defined?

Keep in mind that ACID consistency implies that all writes to a transactional resource must take place as a blocking method call. This seems to be at odds with the concept of asynchronous message passing that works so well with eventual consistency.

However, an application architecture where blocking ACID calls are fundamentally different than asynchronous message passing isn’t really an architecture at all. Developers will have to decide up-front whether or not a given operation is or isn’t synchronous, so the ‘architecture’ offers little implementation guidance. The end result is likely to be a heterogeneous mix of Services, Repositories, Units of Work, Message Channels, etc. A uniform principle will be difficult to distill, and the whole thing threatens to devolve into Spaghetti Code.

The solution turns out to be not at all difficult, but it requires that we invert our thinking a bit. Most of us tend to think about synchronous code first. When we think about code performing synchronous work it seems difficult (perhaps even impossible) to retrofit asynchrony to that model. On the other hand, the converse isn’t true.

Given an asynchronous API, it’s trivial to provide a synchronous, blocking implementation.

Adopting an architecture based on asynchronous message passing (the Pipes and Filters architecture) enables both scenarios. Eventual consistency can be achieved by passing messages around on persistent queues, while ACID consistency can be achieved by handling a message in a blocking call that enlists a (potentially distributed) transaction.

An example seems to be in order here.

Example: keeping score

In the online game world, each player accumulates a score based on his or her actions. From the perspective of the player, the score should always be consistent. When you defeat the zombie boss, you want to see the result in your score right away. That sounds an awful lot like the Player is an Aggregate Root and the score is part of that Entity. ACID consistency is warranted whenever the Player is updated.

On the other hand, each time a score changes it may influence the high score table, but this doesn’t need to be ACID consistent; eventual consistency is fine in this case.

Once again, polymorphism comes to the rescue.

Imagine that the application has a GameEngine class that handles updates in the game. Using an injected IChannel<PointsChangedEvent> it can update the score for a player as simple as this:

            
              /*
Lots of other interesting things happen
            
          

            
                 
* here, like calculating the new score... */
            
          

            
              
                var
              
            
             cmd
=
          

                newScoreChangedEvent(this.playerId,
score);
          

            
              
                this
              
            
            .pointsChannel.Send(cmd);
          

The Send method returns void, so it’s a good example of a naturally asynchronous API. However, the implementation must do two things:

• Update the Player Aggregate Root in a transaction
• Update the high score table (eventually)

That’s two different types of consistency within the same method call.

The first step to enable this is to employ the trusty old Composite design pattern:

            
              
                public
              
            
            
              
                class
              
              
                CompositeChannel
              <T>
: IChannel<T>
          

            {
          

                privatereadonlyIEnumerable<IChannel<T>>
channels;
          

                public CompositeChannel(paramsIChannel<T>[]
channels)
          

               
{
          

                    this.channels
= channels;
          

               
}
          

                publicvoid Send(T
message)
          

               
{
          

                    foreach (var c inthis.channels)
          

                   
{
          

                       
c.Send(message);
          

                   
}
          

               
}
          

            }
          

With a Composite channel it’s possible to compose a polymorphic mix of IChannel<T> implementations, some blocking and some asynchronous.

ACID write

To update the Player Aggregate Root a simple Adapter writes the event to a persistent data store. This could be a relational database, a document database, a REST resource or something else – it doesn’t really matter exactly which technology is used.

            
              
                public
              
            
            
              
                class
              
              
                PlayerStoreChannel
               : 
          

                IChannel<ScoreChangedEvent>
          

            {
          

                privatereadonlyIPlayerStore store;
          

                public PlayerStoreChannel(IPlayerStore store)
          

               
{
          

                    this.store
= store;
          

               
}
          

                publicvoid Send(ScoreChangedEvent message)
          

               
{
          

                    this.store.Save(message.PlayerId,
message);
          

               
}
          

            }
          

The important thing to realize is that the IPlayerStore.Save method will be a blocking method call – perhaps wrapped in a distributed transaction. This ensures that updates to the Player Aggregate Root always leave the data store in a consistent state. Either the operation succeeds or it fails during the method call itself.

This takes care of the ACID consistent write, but the application must also update the high score table.

Asynchronous write

Since eventual consistency is acceptable for the high score table, the message can be transmitted over a persistent queue to be picked up by a background process.

A generic class can server as an Adapter over an IQueue abstraction:

            
              
                public
              
            
            
              
                class
              
              
                QueueChannel
              <T>
: IChannel<T>
          

            {
          

                privatereadonlyIQueue queue;
          

                privatereadonlyIMessageSerializer serializer;
          

                public QueueChannel(IQueue queue,
          

                    IMessageSerializer serializer)
          

               
{
          

                    this.queue
= queue;
          

                    this.serializer
= serializer;
          

               
}
          

                publicvoid Send(T
message)
          

               
{
          

                    this.queue.Enqueue(
          

                        this.serializer.Serialize(message));
          

               
}
          

            }
          

Obvously, the Enqueue method is another void method. In the case of a persistent queue, it’ll block while the message is being written to the queue, but that will tend to be a fast operation.

Composing polymorphic consistency

Now all the building blocks are available to compose both channel implementations into the GameEngine via the CompositeChannel. That might look like this:

            
              
                var
              
            
             playerConnString
= 
            
              ConfigurationManager
            
          

               
.ConnectionStrings["player"].ConnectionString;
          

            
              
                var
              
            
             gameEngine
= newGameEngine(
          

                newCompositeChannel<ScoreChangedEvent>(
          

                    newPlayerStoreChannel(
          

                        newDbPlayerStore(playerConnString)),
          

                    newQueueChannel<ScoreChangedEvent>(
          

                        newPersistentQueue("messageQueue"),                        
          

                        newJsonSerializer())));
          

When the Send method is invoked on the channel, it’ll first invoke a blocking call that ensures ACID consistency for the Player, followed by asynchronous message passing for eventual consistency in other parts of the application.

Conclusion

Even when parts of an application must be implemented in a synchronous fashion to ensure ACID consistency, an architecture based on asynchronous message passing provides a flexible foundation that enables you to polymorphically mix both kinds of consistency in a single method call. From the perspective of the application layer, this provides a consistent and uniform architecture because all mutating actions are modeled as commands end events encapsulated in messages.

Greg Young gave a talk at GOTO Aarhus 2011 titled Developers have a mental disorder, which was (semi-)humorously meant, but still addressed some very real concerns about the cognitive biases of software developers as a group. While I have no intention to provide a complete resume of the talk, Greg said one thing that made me think a bit (more) about SOLID code. To paraphrase, it went something like this:

Developers have a tendency to attempt to solve specific problems with general solutions. This leads to coupling and complexity. Instead of being general, code should be specific.

This sounds correct at first glance, but once again I think that SOLID code offers a solution. Due to the Single Responsibility Principle each SOLID concrete (pardon the pun) class will tend to very specifically address a very narrow problem.

Such a class may implement one (or more) general-purpose interface(s), but the concrete type is specific.

The difference between the generality of an interface and the specificity of a concrete type becomes more and more apparent the better a code base applies the Reused Abstractions Principle. This is best done by defining an API in terms of Role Interfaces, which makes it possible to define a few core abstractions that apply very broadly, while implementations are very specific.

As an example, consider AutoFixture’s ISpecimenBuilder interface. This is a very central interface in AutoFixture (in fact, I don’t even know just how many implementations it has, and I’m currently too lazy to count them). As an API, it has proven to be very generally useful, but each concrete implementation is still very specific, like the CurrentDateTimeGenerator shown here:

            
              
                public
              
            
            
              
                class
              
              
                CurrentDateTimeGenerator
               : 
            
              ISpecimenBuilder
            
          

            {
          

                publicobject Create(object request, 
          

                    ISpecimenContext context)
          

               
{
          

                    if (request
!= typeof(DateTime))
          

                   
{
          

                        returnnewNoSpecimen(request);
          

                   
}
          

                    returnDateTime.Now;
          

               
}
          

            }
          

This is, literally, the entire implementation of the class. I hope we can agree that it’s very specific.

In my opinion, SOLID is a set of principles that can help us keep an API general while each implementation is very specific.

In SOLID code all concrete types are specific.

Soon after I posted my previous blog post on message dispatching without Service Location I received an email from Jeff Saunders with some great observations. Jeff has been so kind to allow me to quote his email here on the blog, so here it is:

“I enjoyed your latest blog post about message dispatching. I have to ask, though: why do we want weakly-typed messages? Why can't we just inject an appropriate IConsumer<T> into our services - they know which messages they're going to send or receive.

“A really good example of this is ISubject<T> from Rx. It implements both IObserver<T> (a message consumer) and IObservable<T> (a message producer) and the default implementation Subject<T> routes messages directly from its IObserver side to its IObservable side.

“We can use this with DI quite nicely - I have written an example in .NET Pad: http://dotnetpad.net/ViewPaste/woTkGk6_GEq3P9xTVEJYZg#c9,c26,

“The good thing about this is that we now have access to all of the standard LINQ query operators and the new ones added in Rx, so we can use a select query to map messages between layers, for instance.

“This way we get all the benefits of a weakly-typed IChannel interface, with the added advantages of strong typing for our messages and composability using Rx.

“One potential benefit of weak typing that could be raised is that we can have just a single implementation for IChannel, instead of an ISubject<T> for each message type. I don't think this is really a benefit, though, as we may want different propagation behaviour for each message type - there are other implementations of ISubject<T> that call consumers asynchronously, and we could pass any IObservable<T> or IObserver<T> into a service for testing purposes.”

These are great observations and I think that Rx holds much promise in this space. Basically you can say that in CQRS-style architectures we’re already pushing events (and commands) around, so why not build upon what the framework offers?

Even if you find the IObserver<T> interface a bit too clunky with its OnNext, OnError and OnCompleted methods compared to the strongly typed IConsumer<T> interface, the question still remains: why do we want weakly-typed messages?

We don’t, necessarily. My previous post wasn’t meant as a particular endorsement of a weakly typed messaging channel. It was more an observation that I’ve seen many variations of this IChannel interface:

            
              
                public
              
            
            
              
                interface
              
            
            
              IChannel
            
          

            {
          

                void Send<T>(T
message);
          

            }
          

The most important thing I wanted to point out was that while the generic type argument may create the illusion that this is a strongly typed method, this is all it is: an illusion. IChannel isn’t strongly typed because you can invoke the Send method with any type of message – and the code will still compile. This is no different than the mechanical distinction between a Service Locator and an Abstract Factory.

Thus, when defining a channel interface I normally prefer to make this explicit and instead model it like this:

            
              
                public
              
            
            
              
                interface
              
            
            
              IChannel
            
          

            {
          

                void Send(object message);
          

            }
          

This achieves exactly the same and is more honest.

Still, this doesn’t really answer Jeff’s question: is this preferable to one or more strongly typed IConsumer<T> dependencies?

Any high-level application entry point that relies on a weakly typed IChannel can get by with a single IChannel dependency. This is flexible, but (just like with Service Locator), it might hide that the client may have (or (d)evolve) too many responsibilities.

If, instead, the client would rely on strongly typed dependencies it becomes much easier to see if/when it violates the Single Responsibility Principle.

In conclusion, I’d tend to prefer strongly typed Datatype Channels instead of a single weakly typed channel, but one shouldn’t underestimate the flexibility of a general-purpose channel either.

Once upon a time I wrote a blog post about why Service Locator is an anti-pattern, and ever since then, I occasionally receive rebuffs from people who agree with me in principle, but think that, still: in various special cases (the argument goes), Service Locator does have its uses.

Most of these arguments actually stem from mistaking the mechanics for the role of a Service Locator. Still, once in a while a compelling argument seems to come my way. One of the most insistent arguments concerns message dispatching – a pattern which is currently gaining in prominence due to the increasing popularity of CQRS, Domain Events and kindred architectural styles.

In this article I’ll first provide a quick sketch of the scenario, followed by a typical implementation based on a ‘Service Locator’, and then conclude by demonstrating why a Service Locator isn’t necessary.

Scenario: Message Dispatching

Appropriate use of message dispatching internally in an application can significantly help decouple the code and make roles explicit. A common implementation utilizes a messaging interface like this one:

            
              
                public
              
            
            
              
                interface
              
            
            
              IChannel
            
          

            {
          

                void Send<T>(T
message);
          

            }
          

Personally, I find that the generic typing of the Send method is entirely redundant (not to mention heavily reminiscent of the shape of a Service Locator), but it’s very common and not particularly important right now (but more about that later).

An application might use the IChannel interface like this:

            
              
                var
              
            
             registerUser
= newRegisterUserCommand(
          

                Guid.NewGuid(),
          

                "Jane
Doe",
          

                "password",
          

                "jane@ploeh.dk");
          

            
              
                this
              
            
            .channel.Send(registerUser);
          

            
              //
...
            
          

            
              
                var
              
            
             changeUserName
= newChangeUserNameCommand(
          

               
registerUser.UserId,
          

                "Jane
Ploeh");
          

            
              
                this
              
            
            .channel.Send(changeUserName);
          

            
              //
...
            
          

            
              
                var
              
            
             resetPassword
= newResetPasswordCommand(
          

               
registerUser.UserId);
          

            
              
                this
              
            
            .channel.Send(resetPassword);
          

Obviously, in this example, the channel variable is an injected instance of the IChannel interface.

On the receiving end, these messages must be dispatched to appropriate consumers, which must all implement this interface:

            
              
                public
              
            
            
              
                interface
              
              
                IConsumer
              <T>
          

            {
          

                void Consume(T
message);
          

            }
          

Thus, each of the command messages in the example have a corresponding consumer:

            
              
                public
              
            
            
              
                class
              
              
                RegisterUserConsumer
               : IConsumer<RegisterUserCommand>
          

            
              
                public
              
            
            
              
                class
              
              
                ChangeUserNameConsumer
               : IConsumer<ChangeUserNameCommand>
          

            
              
                public
              
            
            
              
                class
              
              
                ResetPasswordConsumer
               : IConsumer<ResetPasswordCommand>
          

This certainly is a very powerful pattern, so it’s often used as an argument to prove that Service Locator is, after all, not an anti-pattern.

Message Dispatching using a DI Container

In order to implement IChannel it’s necessary to match messages to their appropriate consumers. One easy way to do this is by employing a DI Container. Here’s an example that uses Autofac to implement IChannel, but any other container would do as well:

            
              
                private
              
            
            
              
                class
              
              
                AutofacChannel
               : 
            
              IChannel
            
          

            {
          

                privatereadonlyIComponentContext container;
          

                public AutofacChannel(IComponentContext container)
          

               
{
          

                    if (container
== null)
          

                        thrownewArgumentNullException("container");
          

                    this.container
= container;
          

               
}
          

                publicvoid Send<T>(T
message)
          

               
{
          

                    var consumer
= this.container.Resolve<IConsumer<T>>();
          

                   
consumer.Consume(message);
          

               
}
          

            }
          

This class is an Adapter from Autofac’s IComponentContext interface to the IChannel interface. At this point I can always see the “Q.E.D.” around the corner: “look! Service Locator isn’t an anti-pattern after all! I’d like to see you implement IChannel without a Service Locator.”

While I’ll do the latter in just a moment, I’d like to dwell on the DI Container-based implementation for a moment.

• Is it simple? Yes.
• Is it flexible? Yes, although it has shortcomings.
• Would I use it like this? Perhaps. It depends :)
• Is it the only way to implement IChannel? No – see the next section.
• Does it use a Service Locator? No.

While AutofacChannel uses Autofac (a DI Container) to implement the functionality, it’s not (necessarily) a Service Locator in action. This was the point I already tried to get across in my previous post about the subject: just because its mechanics look like Service Locator it doesn’t mean that it is one. In my implementation, the AutofacChannel class is a piece of pure infrastructure code. I even made it a private nested class in my Composition Root to underscore the point. The container is still not available to the application code, so is never used in the Service Locator role.

One of the shortcomings about the above implementations is that it provides no fallback mechanism. What happens if the container can’t resolve the matching consumer? Perhaps there isn’t a consumer for the message. That’s entirely possible because there are no safeguards in place to ensure that there’s a consumer for every possibly message.

The shape of the Send method enables the client to send any conceivable message type, and the code still compiles even if no consumer exists. That may look like a problem, but is actually an important insight into implementing an alternative IChannel class.

Message Dispatching using weakly typed matching

Consider the IChannel.Send method once again:

            
              
                void
              
            
             Send<T>(T
message);
          

Despite its generic signature it’s important to realize that this is, in fact, a weakly typed method (at least when used with type inferencing, as in the above example). Equivalently to a bona fide Service Locator, it’s possible for a developer to define a new class (Foo) and send it – and the code still compiles:

            
              
                this
              
            
            .channel.Send(newFoo());
          

However, at run-time, this will fail because there’s no matching consumer. Despite the generic signature of the Send method, it contains no type safety. This insight can be used to implement IChannel without a DI Container.

Before I go on I should point out that I don’t consider the following solution intrinsically superior to using a DI Container. However, readers of my book will know that I consider it a very illuminating exercise to try to implement everything with Poor Man’s DI once in a while.

Using Poor Man’s DI often helps unearth some important design elements of DI because it helps to think about solutions in terms of patterns and principles instead of in terms of technology.

However, once I have arrived at an appropriate conclusion while considering Poor Man’s DI, I still tend to prefer mapping it back to an implementation that involves a DI Container.

Thus, the purpose of this section is first and foremost to outline how message dispatching can be implemented without relying on a Service Locator.

While this alternative implementation isn’t allowed to change any of the existing API, it’s a pure implementation detail to encapsulate the insight about the weakly typed nature of IChannel into a similarly weakly typed consumer interface:

            
              
                private
              
            
            
              
                interface
              
            
            
              IConsumer
            
          

            {
          

                void Consume(object message);
          

            }
          

Notice that this is a private nested interface of my Poor Man’s DI Composition Root – it’s a pure implementation detail. However, given this private interface, it’s now possible to implement IChannel like this:

            
              
                private
              
            
            
              
                class
              
              
                PoorMansChannel
               : 
            
              IChannel
            
          

            {
          

                privatereadonlyIEnumerable<IConsumer>
consumers;
          

                public PoorMansChannel(paramsIConsumer[]
consumers)
          

               
{
          

                    this.consumers
= consumers;
          

               
}
          

                publicvoid Send<T>(T
message)
          

               
{
          

                    foreach (var c inthis.consumers)
          

                       
c.Consume(message);
          

               
}
          

            }
          

Notice that this is another private nested type that belongs to the Composition Root. It loops though all injected consumers, so it’s up to each consumer to decide whether or not to do anything about the message.

A final private nested class bridges the generically typed world with the weakly typed world:

            
              
                private
              
            
            
              
                class
              
              
                Consumer
              <T>
: 
            
              IConsumer
            
          

            {
          

                privatereadonlyIConsumer<T>
consumer;
          

                public Consumer(IConsumer<T>
consumer)
          

               
{
          

                    this.consumer
= consumer;
          

               
}
          

                publicvoid Consume(object message)
          

               
{
          

                    if (message is T)
          

                        this.consumer.Consume((T)message);
          

               
}
          

            }
          

This generic class is another Adapter – this time adapting the generic IConsumer<T> interface to the weakly typed (private) IConsumer interface. Notice that it only delegates the message to the adapted consumer if the type of the message matches the consumer.

Each implementer of IConsumer<T> can be wrapped in the (private) Consumer<T> class and injected into the PoorMansChannel class:

            
              
                var
              
            
             channel
= newPoorMansChannel(
          

                newConsumer<ChangeUserNameCommand>(
          

                    newChangeUserNameConsumer(store)),
          

                newConsumer<RegisterUserCommand>(
          

                    newRegisterUserConsumer(store)),
          

                newConsumer<ResetPasswordCommand>(
          

                    newResetPasswordConsumer(store)));
          

So there you have it: type-based message dispatching without a DI Container in sight. However, it would be easy to use convention-based configuration to scan an assembly and register all IConsumer<T> implementations and wrap them in Consumer<T> instances and use this list to compose a PoorMansChannel instance. However, I will leave this as an exercise to the reader (or a later blog post).

My claim still stands

In conclusion, I find that I can still defend my original claim: Service Locator is an anti-pattern.

That claim, by the way, is falsifiable, so I do appreciate that people take it seriously enough by attempting to disprove it. However, until now, I’ve yet to be presented with a scenario where I couldn’t come up with a better solution that didn’t involve a Service Locator.

Keep in mind that a Service Locator is defined by the role it plays – not the shape of the API.

It’s time to take a step back from the whole debate about whether or not Service Locator is, or isn’t, an anti-pattern. It remains my strong belief that it’s an anti-pattern, while others disagree. Although everyone is welcome to think differently than me, I’ve noticed that some of the arguments being put forth in defense of Service Locator seem very convincing. However, I believe that in those cases we no longer talk about Service Locator, but something that looks an awful lot like it.

Some APIs are easy to confuse with a ‘real’ Service Locator. It probably doesn’t help that last year I published an article on how to tell the difference between a Service Locator and an Abstract Factory. In this article I may have focused too much on the mechanics of Service Locator, but as Derick Bailey was so kind to point out, this hides the role the API might play.

To repeat that earlier post, a Service Locator looks like this:

            
              
                public
              
            
            
              
                interface
              
            
            
              IServiceLocator
            
          

            {
          

               
T Create<T>(object context);
          

            }
          

All Service Locators I’ve seen so far look like that, or some variation thereof, but that doesn’t mean that the relationship is transitive. Just because an API looks like that it doesn’t automatically means that it’s a Service Locator.

If it was, all DI containers would be Service Locators. As an example, here’s Castle Windsor’s Resolve method:

            
              
                public
              
            
             T
Resolve<T>()
          

Even AutoFixture has an API like that:

            
              
                MyClass
              
            
             sut
= fixture.CreateAnonymous<MyClass>();
          

It has never been my intention to denounce every single DI container available, as well as my own open source framework. Service Locator is ultimately not identified by the mechanics of its API, but by the role it plays.

A DI container encapsulated in a Composition Root is not a Service Locator – it’s an infrastructure component.

It becomes a Service Locator if used incorrectly: when application code (as opposed to infrastructure code) actively queries a service in order to be provided with required dependencies, then it has become a Service Locator.

Service Locators are spread thinly and pervasively throughout a code base – that is just as much a defining characteristic.

In my book I describe the Composition Root pattern in chapter 3. This post serves as a summary description of the pattern.

The Constructor Injection pattern is easy to understand until a follow-up question comes up:

Where should we compose object graphs?

It’s easy to understand that each class should require its dependencies through its constructor, but this pushes the responsibility of composing the classes with their dependencies to a third party. Where should that be?

It seems to me that most people are eager to compose as early as possible, but the correct answer is:

As close as possible to the application’s entry point.

This place is called the Composition Root of the application and defined like this:

A Composition Root is a (preferably) unique location in an application where modules are composed together.

This means that all the application code relies solely on Constructor Injection (or other injection patterns), but is never composed. Only at the entry point of the application is the entire object graph finally composed.

The appropriate entry point depends on the framework:

• In console applications it’s the Main method
• In ASP.NET MVC applications it’s global.asax and a custom IControllerFactory
• In WPF applications it’s the Application.OnStartup method
• In WCF it’s a custom ServiceHostFactory
• etc.

(you can read more about framework-specific Composition Roots in chapter 7 of my book.)

The Composition Root is an application infrastructure component.

Only applications should have Composition Roots. Libraries and frameworks shouldn’t.

The Composition Root can be implemented with Poor Man’s DI, but is also the (only) appropriate place to use a DI Container.

A DI Container should only be referenced from the Composition Root. All other modules should have no reference to the container.

Using a DI Container is often a good choice. In that case it should be applied using the Register Resolve Release pattern entirely from within the Composition Root.

Read more in Dependency Injection in .NET.

My recent series of blog posts about Poka-yoke Design generated a few responses (I would have been disappointed had this not been the case). Quite a few of these reactions relate to various serialization or translation technologies usually employed at application boundaries: Serialization, XML (de)hydration, UI validation, etc. Note that such translation happens not only at the perimeter of the application, but also at the persistence layer. ORMs are also a translation mechanism.

Common to most of the comments is that lots of serialization technologies require the presence of a default constructor. As an example, the XmlSerializer requires a default constructor and public writable properties. Most ORMs I’ve investigated seem to have the same kind of requirements. Windows Forms and WPF Controls (UI is also an application boundary) also must have default constructors. Doesn’t that break encapsulation? Yes and no.

Objects at the Boundary

It certainly would break encapsulation if you were to expose your (domain) objects directly at the boundary. Consider a simple XML document like this one:

            
              
                <
              
            
            
              
                name
              
            
            
              >
            
          

            
              
                 
<
              
            
            
              
                firstName
              
              
                >
              Mark</firstName
            
              >
            
          

            
              
                 
<
              
            
            
              
                lastName
              
              
                >
              Seemann</lastName
            
              >
            
          

            
              
                </
              
            
            
              
                name
              
            
            
              >
            
          

Whether or not we have formal contract (XSD) or not, we might stipulate that both the firstName and lastName elements are required. However, despite such a contract, I can easily create a document that breaks it:

            
              
                <
              
            
            
              
                name
              
            
            
              >
            
          

            
              
                 
<
              
            
            
              
                firstName
              
              
                >
              Mark</firstName
            
              >
            
          

            
              
                </
              
            
            
              
                name
              
            
            
              >
            
          

We can’t enforce the contract as there’s no compilation step involved. We can validate input (and output), but that’s a different matter. Exactly because there’s no enforcement it’s very easy to create malformed input. The same argument can be made for UI input forms and any sort of serialized byte sequence. This is why we must treat all input as suspect.

This isn’t a new observation at all. In Patterns of Enterprise Application Architecture, Martin Fowler described this as a Data Transfer Object (DTO). However, despite the name we should realize that DTOs are not really objects at all. This is nothing new either. Back in 2004 Don Box formulated the Four Tenets of Service Orientation. (Yes, I know that they are not in vogue any more and that people wanted to retire them, but some of them still make tons of sense.) Particularly the third tenet is germane to this particular discussion:

Services share schema and contract, not class.

Yes, and that means they are not objects. A DTO is a representation of such a piece of data mapped into an object-oriented language. That still doesn’t make them objects in the sense of encapsulation. It would be impossible. Since all input is suspect, we can hardly enforce any invariants at all.

Often, as Craig Stuntz points out in a comment to one of my previous posts, even if the input is invalid, we want to capture what we did receive in order to present a proper error message (this argument also applies on machine-to-machine boundaries). This means that any DTO must have very weak invariants (if any at all).

DTOs don’t break encapsulation because they aren’t objects at all.

Don’t be fooled by your tooling. The .NET framework very, very much wants you to treat DTOs as objects. Code generation ensues.

However, the strong typing provided by such auto-generated classes gives a false sense of security. You may think that you get rapid feedback from the compiler, but there are many possible ways you can get run-time errors (most notably when you forget to update the auto-generated code based on new schema versions).

An even more problematic result of representing input and output as objects is that it tricks lots of developers into dealing with them as though they represent the real object model. The result is invariably an anemic domain model.

More and more, this line of reasoning is leading me towards the conclusion that the DTO mental model that we have gotten used to over the last ten years is a dead end.

What Should Happen at the Boundary

Given that we write write object-oriented code and that data at the boundary is anything but object-oriented, how do we deal with it?

One option is to stick with what we already have. To bridge the gap we must then develop translation layers that can translate the DTOs to properly encapsulated domain objects. This is the route I take with the samples in my book. However, this is a solution that more and more I’m beginning to think may not be the best. It has issues with maintainability. (Incidentally, that’s the problem with writing a book: at the time you’re done, you know so much more than you did when you started out… Not that I’m denouncing the book – it’s just not perfect…)

Another option is to stop treating data as objects and start treating it as the structured data that it really is. It would be really nice if our programming language had a separate concept of structured data… Interestingly, while C# has nothing of the kind, F# has tons of ways to model data structures without behavior. Perhaps that’s a more honest approach to dealing with data… I will need to experiment more with this…

A third option is to look towards dynamic types. In his article Cutting Edge: Expando Objects in C# 4.0, Dino Esposito outlines a dynamic approach towards consuming structured data that shortcuts auto-generated code and provides a lightweight API to structured data. This also looks like a promising approach… It doesn’t provide compile-time feedback, but that’s only a false sense of security anyway. We must resort to unit tests to get rapid feedback, but we’re all using TDD already, right?

In summary, my entire series about encapsulation relates to object-oriented programming. Although there are lots of technologies available to represent boundary data as ‘objects’, they are false objects. Even if we use an object-oriented language at the boundary, the code has nothing to do with object orientation. Thus, the Poka-yoke Design rules don’t apply there.

Now go back and reread this post, but replace ‘DTO’ with ‘Entity’ (or whatever your ORM calls its representation of a relational table row) and you should begin to see the contours of why ORMs are problematic.

Generics in .NET are wonderful, but sometimes when doing Test-Driven Development against a generic class I’ve felt frustrated because I’ve been feeling that dropping down to the lowest common denominator and testing against, say, Foo<object> doesn’t properly capture the variability inherent in generics. On the other hand, writing the same test for five different types of T have seemed too wasteful (not to mention boring) to bother with.

That’s until it occurred to me that in xUnit.net (and possibly other unit testing frameworks) I can define a generic test class. As an example, I wanted to test-drive a class with this class definition:

            
              
                public
              
            
            
              
                class
              
              
                Interval
              <T>
          

Instead of writing a set of tests against Interval<object> I rather wanted to write a set of tests against a representative set of T. This is so easy to do that I don’t know why I haven’t thought of it before. I simply declared the test class itself as a generic class:

            
              
                public
              
            
            
              
                abstract
              
              
                class
              
              
                IntervalFacts
              <T>
          

The reason I declared the class as abstract is because that effectively prevents the test runner from trying to run the test methods directly on the class. That would fail because T is still open. However, it enabled me to write tests like this:

            [Theory, AutoCatalogData]
          

            
              
                public
              
            
            
              
                void
               MinimumIsCorrect(IComparable<T>
first, 
          

                IComparable<T>
second)
          

            {
          

                var sut
= newInterval<T>(first,
second);
          

                IComparable<T>
result = sut.Minimum;
          

                Assert.Equal(result,
first);
          

            }
          

In this test, I also use AutoFixture’s xUnit.net extension, but that’s completely optional. You might as well just write an old-fashioned unit test, but then you’ll need a SUT Factory that can resolve generics. If you don’t use AutoFixture any other (auto-mocking) container will do, and if you don’t use one of those, you can define a Factory Method that creates appropriate instances of T (and, in this particular case, instances of IComparable<T>).

In the above example I used a [Theory], but I might as well have been using a [Fact] as long as I had a container or a Factory Method to create the appropriate instances.

The above test doesn’t execute in itself because the owning class is abstract. I needed to declare an appropriate set of constructed types for which I wanted the test to run. To do that, I defined the following test classes:

            
              
                public
              
            
            
              
                class
              
              
                DecimalIntervalFacts
               : IntervalFacts<decimal>
{ }
          

            
              
                public
              
            
            
              
                class
              
              
                StringIntervalFacts
               : IntervalFacts<string>
{ }
          

            
              
                public
              
            
            
              
                class
              
              
                DateTimeIntervalFacts
               : IntervalFacts<DateTime>
{ }
          

            
              
                public
              
            
            
              
                class
              
              
                TimSpanIntervalFacts
               : IntervalFacts<TimeSpan>
{ }
          

The only thing these classes do is to each pick a particular type for T. However, since they are concrete classes with test methods, the test runner will pick up the test cases and execute them. This means that the single unit test I wrote above is now being executed four times – one for each constructed type.

It’s even possible to specialize the generic class and add more methods to a single specialized class. As a sanity check I wanted to write a set of tests specifically against Interval<int>, so I added this class as well:

            
              
                public
              
            
            
              
                class
              
              
                Int32IntervalFacts
               : IntervalFacts<int>
          

            {
          

               
[Theory]
          

               
[InlineData(0, 0, 0, true)]
          

               
[InlineData(-1, 1, -1, true)]
          

               
[InlineData(-1, 1, 0, true)]
          

               
[InlineData(-1, 1, 1, true)]
          

               
[InlineData(-1, 1, -2, false)]
          

               
[InlineData(-1, 1, 2, false)]
          

                publicvoid Int32ContainsReturnsCorrectResult(
          

                    int minimum, int maximum, int value,
          

                    bool expectedResult)
          

               
{
          

                    var sut
= newInterval<int>(minimum,
maximum);
          

                    var result
= sut.Contains(value);
          

                    Assert.Equal(expectedResult,
result);
          

               
}
          

                
            
              //
More tests...
            
          

            }
          

When added as an extra class in addition to the four ‘empty’ concrete classes above, it now causes each generic method to be executed five times, whereas the above unit test is only executed for the Int32IntervalFacts class (on the other hand it’s a parameterized test, so the method is actually executed six times).

It’s also possible to write parameterized tests in the generic test class itself:

            [Theory]
          

            [InlineData(-1,
-1, false)]
          

            [InlineData(-1,
0, true)]
          

            [InlineData(-1,
1, true)]
          

            [InlineData(0,
-1, false)]
          

            [InlineData(0,
0, true)]
          

            [InlineData(0,
1, true)]
          

            [InlineData(1,
-1, false)]
          

            [InlineData(1,
0, false)]
          

            [InlineData(1,
1, false)]
          

            
              
                public
              
            
            
              
                void
               ContainsReturnsCorrectResult(
          

                int minumResult, int maximumResult,
          

                bool expectedResult)
          

            {
          

                
            
              //
Test method body
            
          

            }
          

Since this parameterized test in itself has 9 variations and is declared by IntervalFacts<T> which now has 5 constructed implementers, this single test method will be executed 9 x 5 = 45 times!

Not that the the number of executed tests in itself is any measure of the quality of the test, but I do appreciate the ability to write generic unit tests against generic types.

Recently I partook in a Windows Azure migration workshop, helping developers from existing development organizations port their applications to Windows Azure. Once more an old design smell popped up: SQL Server over-utilization. This ought to be old news to anyone with experience designing software on the Wintel stack, but apparently it bears repetition:

Don’t put logic in your database. SQL Server should be used only for persistent storage of data.

(Yes: this post is written in 2011…)

Many years ago I heard that role described as a ‘bit bucket’ – you put in data and pull it out again, and that’s all you do. No fancy stored procedures or functions or triggers.

Why wouldn’t we want to use the database if we have one? Scalability is the answer. SQL Server doesn’t scale horizontally. You can’t add more servers to take the load off a database server (well, some of my old colleagues will argue that this is possible with Oracle, and that may be true, but with SQL Server it’s impossible).

Yes, we can jump through hoops like partitioning and splitting the database up into several smaller databases, but it still doesn’t give us horizontal scalability. SQL Server is a bottleneck in any system in which it takes part.

How is this relevant to Windows Azure? It’s relevant for two important reasons:

• There’s an upper size limit on SQL Azure. Currently that size limit is 50 GB, and while it’s likely to grow in the future, there’s going to be a ceiling for a long time.
• You can’t fine tune the hardware for performance. The server runs on virtual hardware.

Development organizations that rely heavily on the database for execution of logic often need expensive hardware and experienced DBAs to squeeze extra performance out of the database servers. Such people know that write-intensive/append-only tables work best with one type of RAID, while read-intensive tables are better hosted on other file groups on different disks with different RAID configurations.

With SQL Azure you can just forget about all that.

The bottom line is that there are fundamental rules for software development that you must follow if you want to be able to successfully migrate to Windows Azure. I previously described an even simpler sanity check you should perform, but after that you should take a good look at your database.

The best solution is if you can completely replace SQL Server with Azure’s very scalable storage services, but those come with their own set of challenges.

Rapid feedback is one of the cornerstones of agile development. The faster we can get feedback, the less it costs to correct errors. Unit testing is one of the ways to get feedback, but not the only one. Each way we can get feedback comes with its own advantages and disadvantages.

In my experience there’s a tradeoff between the cost of setting up a feedback mechanism and the level of confidence we can get from it. This is quite intuitive to most people, but I’ve rarely seen it explicitly described. The purpose of this post is to explicitly describe and relate those different mechanisms.

Compilation

In compiled languages, compilation provides the first level of feedback. I think a lot of people don’t think about this as feedback as (for compiled languages) it’s a necessary step before the code can be executed.

The level of confidence may not be very high – we tend to laugh at statements like ‘if it compiles, it works’. However, the compiler still catches a lot of mistakes. Think about it: does your code always compile, or do you sometimes get compilation errors? Do you sometimes try to compile your code just to see if it’s possible? I certainly do, and that’s because the compiler is always available, and it’s the first and fastest level of feedback we get.

Code can be designed to take advantage of this verification step. Constructor Injection, for example, ensures that we can’t create a new instance of a class without supplying it with its required dependencies.

It’s also interesting to note that anecdotal evidence seems to suggest that unit testing is more prevalent for interpreted languages. That makes a lot of sense, because as developers we need feedback as soon as possible, and if there’s no compiler we must reach for the next level of feedback mechanism.

Static code analysis

The idea of static code analysis isn’t specific to .NET, but in certain versions of Visual Studio we have Code Analysis (which is also available as FxCop). Static code analysis is a step up from compilation – not only is code checked for syntactical correctness, but it’s also checked against a set of known anti-patterns and idioms.

Static code analysis is encapsulated expert knowledge, yet surprisingly few people use it. I think part of the reason for this is because the ratio of time against confidence isn’t as compelling as with other feedback mechanism. It takes some time to run the analysis, but like compilation the level of confidence gained from getting a clean result is not very high.

Personally, I use static code analysis once in a while (e.g. before checking in code to the default branch), but not as often as other feedback mechanisms. Still, I think this type of feedback mechanism is under-utilized.

Unit testing

Running a unit test suite is one of the most efficient ways to gain rapid feedback. The execution time is often comparable to running static code analysis while the level of confidence is much higher.

Obviously a successful unit test execution is no guarantee that the final application works as intended, but it’s a much better indicator than the two previous mechanisms. When presented with the choice of using either static code analysis or unit tests, I prefer unit tests every time because the confidence level is much higher.

If you have sufficiently high code coverage I’d say that a successful unit test suite is enough confidence to check in code and let continuous integration take care of the rest.

Keep in mind that continuous integration and other types of automated builds are feedback mechanisms. If you institute rules where people are ‘punished’ for checking in code that breaks the build, you effectively disable the automated build as a source of feedback.

Thus, the rest of these feedback mechanisms should be used rarely (if at all) on developer work stations.

Integration testing

Integration testing comes in several sub-categories. You can integrate multiple classes from within the same library, or integrate multiple libraries while still using Test Doubles instead of out-of-process resources. At this level, the cost/confidence ratio is comparable to unit testing.

Subcutaneous testing

A more full level of integration testing comes when all resources are integrated, but the tests are still driven by code instead of through the UI. While this gives a degree of confidence that is close to what you can get from a full systems test, the cost of setting up the entire testing environment is much higher. In many cases I consider this the realm of a dedicated testing department, as it’s often a full-time job to maintain the suite of automation tools that enable such tests – particularly if we are talking about distributed systems.

System testing

This is a test of the full system, often driven through the UI and supplemented with manual testing (such as exploratory testing). This provides the highest degree of confidence, but comes with a very high cost. It’s often at this level we find formal acceptance tests.

The time before we can get feedback at this level is often considerable. It may take days or even weeks or months.

Understand the cost/confidence ratio

The purpose of this post has been to talk about different ways we can get feedback when developing software. If we could get instantaneous feedback with guaranteed confidence it would be easy to choose, but as this is not the case we must understand the benefits and drawbacks of each mechanism.

It’s important to realize that there are many mechanisms, and that we can combine them in a staggered approach where we start with fast feedback (like the compiler) with a low degree of confidence and then move through successive stages that each provide a higher level of confidence.

Developers exposed to ASP.NET are likely to be familiar with the so-called Provider pattern. You see it a lot in that part of the BCL: Role Provider, Membership Provider, Profile Provider, etc. Lots of text has already been written about Providers, but the reason I want to add yet another blog post on the topic is because once in a while I get the question on how it relates to Dependency Injection (DI).

Is Provider a proper way to do DI?

No, it has nothing to do with DI, but as it tries to mimic loose coupling I can understand the confusion.

First things first. Let’s start with the name. Is it a pattern at all? Regular readers of this blog may get the impression that I’m fond of calling everything and the kitchen sink an anti-pattern. That’s not true because I only make that claim when I’m certain I can hold that position, so I’m not going to denounce Provider as an anti-pattern. On the contrary I will make the claim that Provider is not a pattern at all.

A design pattern is not invented – it’s discovered as a repeated solution to a commonly recurring problem. Providers, on the other hand, were invented by Microsoft, and I’ve rarely seen them used outside their original scope. Secondly I’d also dispute that they solve anything.

That aside, however, I want to explain why Provider is bad design:

• It uses the Constrained Construction anti-pattern
• It hides complexity
• It prevents proper lifetime management
• It’s not testable

In the rest of this post I will explain each point in detail, but before I do that we need an example to look at. The old OrderProcessor example suffices, but instead of injecting IOrderValidator, IOrderCollector, and IOrderShipper this variation uses Providers to provide instances of the Services:

            
              
                public
              
            
            
              
                SuccessResult
               Process(Order order)
          

            {
          

                IOrderValidator validator
= 
          

                    ValidatorProvider.Validator;
          

                bool isValid
= validator.Validate(order);
          

                if (isValid)
          

               
{
          

                    CollectorProvider.Collector.Collect(order);
          

                    ShipperProvider.Shipper.Ship(order);
          

               
}
          

                returnthis.CreateStatus(isValid);
          

            }
          

The ValidatorProvider uses the configuration system to create and return an instance of IOrderValidator:

            
              
                public
              
            
            
              
                static
              
              
                IOrderValidator
               Validator
          

            {
          

                get
          

               
{
          

                    var section
= 
          

                        
            
              OrderValidationConfigurationSection
            
          

                           
.GetSection();
          

                    var typeName
= section.ValidatorTypeName;
          

                    var type
= Type.GetType(typeName, true);
          

                    var obj
= Activator.CreateInstance(type);
          

                    return (IOrderValidator)obj;
          

               
}
          

            }
          

There are lots of details I omitted here. I could have saved the reference for later use instead of creating a new instance each time the property is accessed. In that case I would also have had to make the code thread-safe, so I decided to skip that complexity. The code could also be more defensive, but I’m sure you get the picture.

The type name is defined in the app.config file like this:

            
              
                <
              
            
            
              
                orderValidation
              
            
            
              
              
            
          

            
              
                  
              
            
            
              
                type
              
              
                =
              "
            
              Ploeh.Samples.OrderModel.UnitTest.TrueOrderValidator,
            
          

            
              
                       
Ploeh.Samples.OrderModel.UnitTest
              
            
            "
            
               />
            
          

Obviously, CollectorProvider and ShipperProvider follow the same… blueprint.

This should be well-known to most .NET developers, so what’s wrong with this model?

Constrained Construction

In my book’s chapter on DI anti-patterns I describe the Constrained Construction anti-pattern. Basically it occurs every time there’s an implicit constraint on the constructor of an implementer. In the case of Providers the constraint is that each implementer must have a default constructor. In the example the culprit is this line of code:

            
              
                var
              
            
             obj
= Activator.CreateInstance(type);
          

This constrains any implementation of IOrderValidator to have a default constructor, which obviously means that the most fundamental DI pattern Constructor Injection is out of the question.

Variations of the Provider idiom is to supply an Initialize method with a context, but this creates a temporal coupling while still not enabling us to inject arbitrary Services into our implementations. I’m not going to repeat six pages of detailed description of Constrained Construction here, but the bottom line is that you can’t fix it – you have to refactor towards true DI – preferably Constructor Injection.

Hidden complexity

Providers hide the complexity of their implementations. This is not the same as encapsulation. Rather it’s a dishonest API and the problem is that it just postpones the moment when you discover how complex the implementation really is.

When you implement a client and use code like the following everything looks deceptively simple:

            
              
                IOrderValidator
              
            
             validator
= 
          

                ValidatorProvider.Validator;
          

However, if this is the only line of code you write it will fail, but you will not notice until run-time. Check back to the implementation of the Validator property if you need to refresh the implementation: there’s a lot of things that can go wrong here:

• The appropriate configuration section is not available in the app.config file.
• The ValidatorTypeName is not provided, or is null, or is malformed.
• The ValidatorTypeName is correctly formed, but the type in question cannot be located by Fusion.
• The Type doesn’t have a default constructor. This is one of the other problems of Constrained Construction: it can’t be statically enforced because a constructor is not part of an abstraction’s API.
• The created type doesn’t implement IOrderValidator.

I’m sure I even forgot a thing or two, but the above list is sufficient for me. None of these problems are caught by the compiler, so you don’t discover these issues until you run an integration test. So much for rapid feedback.

I don’t like APIs that lie about their complexity.

Hiding complexity does not make an API easier to use; it makes it harder.

An API that hides necessary complexity makes it impossible to discover problems at compile time. It simply creates more friction.

Lifetime management issues

A Provider exerts too much control over the instances it creates. This is a variation of the Control Freak anti-pattern (also from my book). In the current implementation the Validator property totally violates the Principle of least surprise since it returns a new instance every time you invoke the getter. I did this to keep the implementation simple (this is, after all, example code), but a more normal implementation would reuse the same instance every time.

However, reusing the same instance every time may be problematic in a multi-threaded context (such as a web application) because you’ll need to make sure that the implementation is thread-safe. Often, we’d much prefer to scope the lifetime of the Service to each HTTP request.

HTTP request scoping can be built into the Provider, but then it would only work in web applications. That’s not very flexible.

What’s even more problematic is that once we move away from the Singleton lifestyle (not to be confused with the Singleton design pattern) we may have a memory leak at hand, since the implementation may implement IDisposable. This can be solved by adding a Release method to each Provider, but now we are moving so far into DI Container territory that I find it far more reasonable to just use proper DI instead of trying to reinvent the wheel.

Furthermore, the fact that each Provider owns the lifetime of the Service it controls makes it impossible to share resources. What if the implementation we want to use implements several Role Interfaces each served up by a different Provider? We might want to use that common implementation to share or coordinate state across different Services, but that’s not possible because we can’t share an instance across multiple providers.

Even if we configure all Providers with the same concrete class, each will instantiate and serve its own separate instance.

Testability

The Control Freak also impacts testability. Since a Provider creates instances of interfaces based on XML configuration and Activator.CreateInstance, there’s no way to inject a dynamic mock.

It is possible to use hard-coded Test Doubles such as Stubs or Fakes because we can configure the XML with their type names, but even a Spy is problematic because we’ll rarely have an object reference to the Test Double.

In short, the Provider idiom is not a good approach to loose coupling. Although Microsoft uses it in some of their products, it only leads to problems, so there’s no reason to mimic it. Instead, use Constructor Injection to create loosely coupled components and wire them in the application’s Composition Root using the Register Resolve Release pattern.

When AutoFixture creates complex objects it invokes the constructors of the types in question. When a class exposes more than one constructor, the default behavior is to pick the constructor with the fewest number  of arguments. This is the exact opposite of most DI Containers that select the greediest constructor.

For a DI Container it makes more sense to select the greediest constructor because that maximizes the chance that all dependencies are properly being auto-wired.

The reason that AutoFixture’s default behavior is different is that it maximizes the chance that an object graph can be created at all. If a convenience overload is available, this gives AutoFixture a better chance to compose the object graph.

This works well until a class comes along and introduces the wrong kind of ambiguity. Consider this case of Bastard Injection (Dependency Injection in .NET, section 5.2):

            
              
                public
              
            
            
              
                class
              
            
            
              Bastard
            
          

            {
          

                privatereadonlyIFoo foo;
          

                public Bastard()
          

                   
: this(newDefaultFoo())
          

               
{
          

               
}
          

                public Bastard(IFoo foo)
          

               
{
          

                    if (foo
== null)
          

                   
{
          

                        thrownewArgumentNullException("foo");
          

                   
}
          

                    this.foo
= foo;
          

               
}
          

                publicIFoo Foo
          

               
{
          

                    get { returnthis.foo;
}
          

               
}
          

            }
          

By default AutoFixture will use the default constructor which means that the Foo will always be the instance of DefaultFoo owned by the Bastard class itself. This is problematic if we wish to inject and freeze a Test Double because even if we freeze an IFoo instance, it will never be injected because the default constructor is being invoked.

            
              
                var
              
            
             fixture
= newFixture();
          

            fixture.Register<IFoo>(
          

               
fixture.CreateAnonymous<DummyFoo>);
          

            
              
                var
              
            
             b
= fixture.CreateAnonymous<Bastard>();
          

            
              
                Assert
              
            
            .IsAssignableFrom<DefaultFoo>(b.Foo);
          

As the above unit test demonstrates, even though the Register method call defines a mapping from IFoo to DummyFoo, the Foo property is an instance of DefaultFoo. The DummyFoo instance is never injected into the Bastard instance since the default constructor is used.

Your first reaction in such a case should be to get rid of all convenience constructors to make the the class’ constructor unambiguous. However, it’s also possible to change AutoFixture’s behavior.

The following is a description of a feature that will be a available in AutoFixture 2.1. It’s not available in AutoFixture 2.0, but is already available in the code repository. Thus, if you can’t wait for AutoFixture 2.1 you can download the source and build it.

As always, AutoFixture’s very extensible interface makes it possible to change this behavior. Constructor selection is guided by an interface called IConstructorQuery, and while ModestConstructorQuery is the default implementation, there’s also an implementation called GreedyConstructorQuery.

To change the behavior specifically for the Bastard class the Fixture instance must be customized. The following unit test demonstrates how to do that.

            
              
                var
              
            
             fixture
= newFixture();
          

            fixture.Customize<Bastard>(c
=> c.FromFactory(
          

                newConstructorInvoker(
          

                    newGreedyConstructorQuery())));
          

            fixture.Register<IFoo>(
          

               
fixture.CreateAnonymous<DummyFoo>);
          

            
              
                var
              
            
             b
= fixture.CreateAnonymous<Bastard>();
          

            
              
                Assert
              
            
            .IsAssignableFrom<DummyFoo>(b.Foo);
          

Notice that the only difference from before is an additional call to the Customize method where Bastard is modified to use greedy constructor selection. This changes the factory for the Bastard type, but not for any other types.

If you’d rather prefer to completely change the overall constructor selection behavior for AutoFixture you can add the GreedyConstructorQuery wrapped in a ConstructorInvoker to the Fixture’s Customizations:

            
              
                var
              
            
             fixture
= newFixture();
          

            fixture.Customizations.Add(
          

                newConstructorInvoker(
          

                    newGreedyConstructorQuery()));
          

            fixture.Register<IFoo>(
          

               
fixture.CreateAnonymous<DummyFoo>);
          

            
              
                var
              
            
             b
= fixture.CreateAnonymous<Bastard>();
          

            
              
                Assert
              
            
            .IsAssignableFrom<DummyFoo>(b.Foo);
          

In this unit test, the only change from the previous is that instead of assigning the GreedyConstructorQuery specifically to Bastard, it’s now assigned as the new default strategy. All specimens created by AutoFixture will be created by invoking their most greedy constructor.

As always I find the default behavior more attractive, but the option to change behavior is there if you need it.