REST lesson learned: Consider a home link on all resources

Thursday, 02 May 2013 10:51:00 UTC

Add home links on RESTful resources.

This suggestion is part of my REST lessons learned series of blog posts. Contrary to to the previous posts, this advice doesn't originate from a lesson learned the hard way, but is more of a gentle suggestion.

When designing a level 3 RESTful API, I've found that it's often helpful to think about design issues in terms of: how would you design it if it was a web site? On web sites, we don't ask users to construct URLs and enter them into the browser's address bar. On web sites, users follow links (and fill out forms). Many well-designed web sites are actually HATEOAS services, so it makes sense to learn from them when designing RESTful APIs.

One almost universal principle for well-designed web sites is that they always have a 'home' link (usually in the top left corner). It makes sense to transfer this principle to RESTful APIs.

All the RESTful APIs that I've helped design so far have had a single 'home' resource, which acts as a starting point for all clients. This is the only published URL for the entire API. From that starting page, clients must follow (semantic) links in order to accomplish their goals.

As an example, here's a 'home' resource on a hypothetical product catalog API:

<home xmlns="http://fnaah.ploeh.dk/productcatalog/2013/05"
      xmlns:atom="http://www.w3.org/2005/Atom">
  <atom:link href="http://catalog.api.ploeh.dk/products/1234"
             rel="http://catalog.api.ploeh.dk/docs/rels/products/discounted" />
  <atom:link href="http://catalog.api.ploeh.dk/products/5678"
             rel="http://catalog.api.ploeh.dk/docs/rels/products/discounted" />
  <atom:link href="http://catalog.api.ploeh.dk/products/1337"
             rel="http://catalog.api.ploeh.dk/docs/rels/products/discounted" />
  <atom:link href="http://catalog.api.ploeh.dk/categories"
             rel="http://catalog.api.ploeh.dk/docs/rels/product/categories" />
</home>

From this (rather sparse) 'home' resource, you can view each discounted product by following the discounted link, or you can browse the catalog by following the categories link.

If you follow one of the discounted links, you get a new resource:

<product xmlns="http://fnaah.ploeh.dk/productcatalog/2013/05"
         xmlns:atom="http://www.w3.org/2005/Atom">
  <atom:link href="http://catalog.api.ploeh.dk"
             rel="http://catalog.api.ploeh.dk/docs/rels/home" />
  <atom:link href="http://catalog.api.ploeh.dk/categories/chocolate"
             rel="http://catalog.api.ploeh.dk/docs/rels/category" />
  <atom:link href="http://catalog.api.ploeh.dk/categories/gourmet"
             rel="http://catalog.api.ploeh.dk/docs/rels/category" />
  <name>Fine Criollo dark chocolate</name>
  <price>50 DKK</price>
  <weight>100 g</weight>
</product>

Notice that in addition to the the two category links, the resource representation also contains a home link, enabling the client to go back to the 'home' resource.

Having a home link on all resources is another courtesy to the client, just like avoiding 204 responses - in fact, if you can't think of anything else to return instead of 204 (No Content), at the very least, you could return a home link.

In the above examples, I didn't obfuscate the URLs, but I would do that for a real REST API. The reason I didn't obfuscate the URLs here is to make the example easier to understand.


REST lesson learned: Avoid hackable URLs

Wednesday, 01 May 2013 10:29:00 UTC

Avoid hackable URLs if you are building a HATEOAS API.

This is a lesson about REST API design that I learned while building non-trivial REST APIs. If you provide a full-on level 3 REST API, consider avoiding hackable URLs.

Hackable URLs #

A hackable URL is a URL where there's a clear pattern or template for constructing the URL. As an example, if I present to you the URL http://foo.ploeh.dk/products/1234, it's easy to guess that this is a resource representing a product with the SKU of 1234. If you know the SKU of another product, it's easy to 'hack' the URL to produce e.g. http://foo.ploeh.dk/products/5678.

That's a really nice feature of your API if you are doing a level 1 or 2 API, but for a HATEOAS API, this defies the purpose.

The great divide #

Please notice that the shift from level 2 to level 3 RESTful APIs mark a fundamental shift in the way you should approach URL design.

Hackable URLs are great for level 1 and 2 APIs because the way you (as a client) are told to construct URLs is by assembling them from templates. As an example, the Windows Azure REST APIs explicitly instruct you to construct the URL in a particular way: the URL to get BLOB container properties is https://myaccount.blob.core.windows.net/mycontainer?restype=container, where you should replace myaccount with your account name, and mycontainer with your container name. While code aesthetics are subjective, it's not even a particularly clean URL, but it's easy enough to produce. The URL template is part of the contract, which puts the Windows Azure API solidly at level 2 of the Richardson Maturity Model. If I were designing a level 1 or 2 API, I'd make sure to make URLs hackable, too.

However, if you're building a level 3 API, hypermedia is king. Clients are expected to follow links. The addresses of resources are not published as having a particular template; instead, clients must follow semantic links in order to arrive at the desired resource(s). When hypermedia is the engine of application state, it's no good if the client can short-circuit the application flow by 'hacking' URLs. It may leave the application in an inconsistent state if it tries to do that.

Hackable URLs are great for level 1 and 2 APIs, but counter-productive for level 3 APIs.

Evolving URLs #

One of the main attractions of building a level 3 RESTful API is that it's easier to evolve. Exactly because URL templates are not part of the contract, you can decide to change the URL structure when evolving your API.

Imagine that the first version of your API has an (internal) URL template like /orders/{customerId}, so that the example URL http://foo.ploeh.dk/orders/1234 is the address of the order history for customer 1234. However, in version 2 of your API, you realize that this way of thinking is still too RPC-like, and you'd rather prefer /customers/{customerId}/order, e.g. http://foo.ploeh.dk/customers/1234/orders.

With a level 3 RESTful API, you can change your internal URL templates, and as long as you keep providing links, clients following links will not notice the difference. However, if clients are 'hacking' your URLs, their applications may stop working if you change URL templates.

Keep clients safe #

In the end, HATEOAS is about encapsulation: make it easy for the client to do the right thing, and make it hard for the client to do the wrong thing. Following links will make clients more robust, because they will be able to handle changes in the API. Making it easy for clients to follow links is one side of designing a good API, but the other side is important too: make it difficult for clients to not follow links: make it difficult for clients to 'hack' URLs.

The services I've helped design so far are level 3 APIs, but they still used hackable URLs. One reason for that was that this is the default in the implementation platform we used (ASP.NET Web API); another reason was that I thought it would be easier for me and the rest of the development team if the URLs were human-readable. Today, I think this decision was a mistake.

What's the harm of supplying human-readable URLs for a level 3 RESTful API? After all, if a client only follows links, the values of the URLs shouldn't matter.

Indeed, if the client only follows links. However, clients are created by human developers, and humans often take the road of least resistance. While there are long-time benefits (robustness) from following links, it is more work in the short term. The API team and I repeatedly experienced that the developers consuming our APIs had 'hacked' our URLs; when we changed our URL templates, their clients broke and they complained. Even though we had tried to explicitly tell them that they must follow links, they didn't. While we never documented our URL templates, they were simply too easy to guess from pure extrapolation.

Opaque URLs #

In the future, I plan to make URLs opaque when building level 3 APIs. Instead of http://foo.ploeh.dk/customers/1234/orders, I'm going to make it http://foo.ploeh.dk/DC884298C70C41798ABE9052DC69CAEE, and instead of http://foo.ploeh.dk/products/2345, I'm going to make it http://foo.ploeh.dk/598CB0CAC30646E1BB768596BFE91F2C, and so on.

Obviously, that means that my API will have to maintain some sort of two-way lookup table that can map DC884298C70C41798ABE9052DC69CAEE to a request for customer 1234's orders, 598CB0CAC30646E1BB768596BFE91F2C to request for product 2345, but that's trivial to implement.

It puts a small burden on the server(s), but effectively stops client developers from shooting themselves in their feet.

Summary #

Hackable URLs are a good idea if you are building a web site, or a level 1 or 2 REST API, but unless you know that all client developers are enthusiastic RESTafarians, consider producing opaque URLs for level 3 REST APIs.


Comments

I understand the benefits of the idea that clients must follow links, I just can't see how it would work in reality. Say you are building a website to display products and you are consuming a rest API, how do you support deep linking to individual products? OK, you can either use the same URL as the underlying API or encode it in, but what are you expected to do if the API changes it's links? Redirect the user to your home page? What if you want to display information from 2 or more resources on the same webpage?

2013-05-15 8:04 UTC

First of all, keep in mind that while it can be helpful to think about REST design principles in terms of "how would I design this if it was a web site", a REST API is not a web site.

You can do deep linking in your web site, but why would you want to do 'deep linking' for an API? These are two different concerns.

It's very common to create a web site where each page calls many individual services. This can be done either from the web server (e.g. from ASP.NET or similar), or from the browser as AJAX calls. This is commonly known as mash-up architecture, because the GUI is really just a mash-up of service data. Amazon.com works that way. You can still deep link to a web page; it's the web page's responsibility to figure out which services to call with what parameters.

That said, as described in the RESTful Web Services Cookbook, you should serve cool URLs, so if you ever decide to change your internal URI template, you should leave a 301 (Moved Permanently) behind at the old URL. This would enable a client that once bookmarked a resource to follow the redirect to the new address.

2013-05-15 9:11 UTC

When you say "it's the web page's responsibility to figure out which services to call", that's what I'm getting at, how would the client do that?

I'm guessing one way would be to cache links followed and update them on 301s or "re-follow" on 404s. So, say you wanted a web page displaying "product/24", you might:

  • GET the REST endpoint
  • GET product catalogue URL from response
  • GET product URL from response
  • Cache the product URL against the website product URL
  • Subsequent requests hit the cached URL

Then if the product URL changes, if the response is 301, you simply update the cache. If the response is 404 then you'd redo the above steps.

I'm just thinking this through, is the above a "standard" approach for creating rest clients?

2013-05-15 12:04 UTC

That sounds like one way of doing it. I don't think there's a 'standard' way for creating RESTful clients.

The sketch you paint sounds like a lot of work, and it seems that it would be easier if the client could simply assemble appropriate URLs from templates. That would indicate level 1 and 2 REST APIs, which might be perfectly fine if the only purpose of building the service is to support a GUI. However, what you get in exchange for the extra effort it takes to consume a level 3 API, is better decoupling. It's always going to be a trade-off.

It's definitely going to be more work to build and consume a truly HATEOAS-based API, so you should only do it if it's going to provide a good return on investment. When would that be? One general scenario I can think of is when you're building a service, which is going to be consumed by multiple (unknown) clients. If you control the service, but not the clients, I'd say a level 3 API is very beneficial, because it enables you to evolve the API independently of the clients. Conversely, if the only purpose of building a service is to support a single client, it's probably going to be overkill.

2013-05-15 15:15 UTC

The way I see it is that being HATEOAS compliant does not impose non-hackable URLs, but that URLs - even if they might look hackable (/1, /2...etc) - are not guaranteed to work, are not part of the contract and thus should not be relied uppon. In other words, a link is only guaranteed to work if you, the client, got it from a previous reponse, be it "hackable-looking" or not.

So I think opaque URLs are just a way to inforce that contract. But don't we always here that "a good REST client should be well behaved" ?

2014-10-10 11:11 UTC

Reda, thank you for writing. You're right, and if we could rely on all clients to be well-behaved, there'd be no problems. However, in my experience, client developers often don't read the documentation particularly thoroughly. Instead, they look at the returned data and start inferring the URL scheme from examples. I already wrote about this in this post:

"The API team and I repeatedly experienced that the developers consuming our APIs had 'hacked' our URLs; when we changed our URL templates, their clients broke and they complained. Even though we had tried to explicitly tell them that they must follow links, they didn't."
So, in this imperfect world, non-hackable URLs start to look attractive, because then the client developers have no choice but to follow the links.

2014-10-10 12:02 UTC

I'd like to propose an alternate strategy to achieve the desired result while maintaining the ability to easily debug production issues at the client: only use the non-hackable urls for the "dev sandbox".

Another inducement to better client behvior might be to have the getting started documentation use existing libraries for the chosen hypermedia media type.

Finally, have the clients send a "developer [org] identifier". Find a way to probe clients for correctness. For instance, occasionally alter hrefs, but also support the "hackable" form.

With this data, be proactive about informing the user that "we're planning a release and your app will probably break because you're not following links as expected".

In the end, some people will still fail and be angry. With opaque urls, I would worry that users would have a hard time communicating production issues when looking at logs /fiddler / browser tools / etc.

2014-12-23 17:40 UTC

Kijana, thank you for writing. You suggest various good ideas that I'll have to keep in mind. Not all of them are universally applicable, but then I also don't think that my suggestion should be applied indiscriminately.

Your first suggestion assumes that there is a dev sandbox; this may or may not be the case. The APIs with which I currently work have no dev sandboxes.

The idea about using existing client libraries for the chosen hypermedia type is only possible if such libraries exist. The APIs I currently design use vendor media types, so client libraries only exist if we develop them ourselves. However, one of the important goals of RESTful services is to ensure interoperability, so we can't assume that clients are going to run on .NET, or Java, or Ruby, or whatever. For vendor media types, I don't think this is a viable option.

Using a client identifier is an option, but in order to work well, there must be some way to correlate that identifier to a contact in the client's organisation. That's an option if you already have a mechanism in place where you only allow known clients to access your API. On the other hand, if you publish a truly scalable public API, you may not want to do that, as registration requirements tend to hurt adoption. The APIs I currently work with is a mix of both of those; some are publicly accessible, while others require a 'client key'.

These are all interesting ideas, but ultimately, I'm not sure I understand your concern. Let's first establish that 'users' are other programmers. Why would a URL like http://foo.ploeh.dk/DC884298C70C41798ABE9052DC69CAEE be harder to communicate than http://foo.ploeh.dk/customers/1234/orders? Isn't it copy and paste in both cases?

To be clear: I don't claim that obfuscated URLs don't make client developers' work more difficult; it does, compared to 'cheating' by hacking the URL schemes, instead of following links. Even for well-behaved client developers, another level of abstraction always makes things harder. On the other hand, this should also make clients more robust, so as always, it's a trade-off.

2015-01-22 08:55 UTC

“The API team and I repeatedly experienced that the developers consuming our APIs had 'hacked' our URLs; when we changed our URL templates, their clients broke and they complained. Even though we had tried to explicitly tell them that they must follow links, they didn't.”

I recently worked on an API for an iOS app and we ran into the same problem. It didn't matter how often I said "these URLs are probably going to change, don't hard-code them into the app" we still ran into issues. This was an internal team, I can only imagine how much more difficult it would be with an external team.

Instead of hashing the URLs, we changed the server-side URL generation to HMAC the URL and append the signature onto the query string. Requests without a valid signature would return a 403 Forbidden response. The root of the domain is the only URL that doesn't require a signature, and it returns a json response with signed URLs for each of the "base" resources.

We enabled this on a staging server and since all dev was happening on it, all the hard-coded URLs stopped functioning and were quickly removed from the application. It forced both us and the iOS developers to think more about navigation between resources, since it was impossible to get from one resource to another without valid URLs included in the json responses. I think it was probably also nicer for debugging than hashed URLs would be.

2015-04-11 09:26 UTC

Dan, thank you for sharing that great idea! That looks like a much better solution than my original suggestion of obscuring the URL, because the URL is still human-readable, and thus probably still easier to troubleshoot for developers.

Your suggestion also doesn't require a lookup table. My suggested solution would require a lookup table in order to understand what each obscured URL actually means, and if you're running on multiple servers, that lookup table would have to be kept consistent across all servers, which can be hard to do in itself (you can use a database, but then you'd have a single point of failure). Your solution doesn't need any of that; it only requires that the HMAC key is the same on all servers.

The only disadvantage that I can think of is that you may need to keep that HMAC key secret, particularly in internal projects, in order to prevent clients from (literally) hacking the URLs.

Still, it sounds like your solution has more advantages, so I'm going to try that approach next time. Thank you for sharing!

2015-04-11 10:21 UTC

“The only disadvantage that I can think of is that you may need to keep that HMAC key secret, particularly in internal projects, in order to prevent clients from (literally) hacking the URLs.”

That was one concern of ours too. One approach we considered was including the git commit SHA as part of the HMAC secret so that every commit would invalidate existing URLs. We decided against it because we were doing Continous Delivery, so multiple times a day we were deploying to the server, and we didn't want the communication overhead of having to notify everyone on each commit. We wanted to be able to merge feature branches into master and just know the CI server was going to handle deployment; and that unless we got alerts from CI, Pingdom or New Relic, everything is working as expected. The last thing we wanted to do was babysit the build so we could tell everyone "ok, the new build has deployed to staging, URLs will break so restart any clients you are in the middle of testing". Also part of the reason we use REST (and json-api) was to decouple front and backend development, it seemed counterproductive to introduce coupling back into the process.

An internal team could still re-implement this whole mechanism and sign their own URLs. We could either generate random secrets every day or have a simple tool that we can use to change the secret at-will (probably both).

Even though it doesn't completely stop URL hard coding, another idea we had was to include an expiration timestamp in the query string and then HMAC the URL with the timestamp included. We would use the responses' Cache-Control max-age to calculate the time, which would allow the client to use any URLs from cached responses. If the client didn't implement proper cache invalidation the URLs would inexplicably break, thus having a nice side-effect of forcing the client to handle Cache-Control max-age properly.

2015-04-11 17:58 UTC

Dan, there are lots of interesting ideas there. What I assumed from your first comment is that you'd calculate the HMAC using asymmetric encryption, which would mean that unless clients have the server's encryption key, they wouldn't be able to recalculate the HMAC, and that would effectively prevent them from attempting to 'guess' URLs instead of following links.

If an external client has the server's encryption key, you have a different sort of problem.

However, for internal clients, developers may actually be able to find the key in your source control repository, build server, or wherever else you keep it (depending on the size of your organisation). You can solve this with security measures, like ACL-based security on the crypto key itself. It not hard, but it's something you may explicitly have to do.

When it comes to tying the URL to cache invalidation, that makes me a bit uneasy. While RESTful clients should follow links, they are allowed to bookmark links. It's a fundamental tenet of proper web design (not only REST) that cool URIs don't change. A client should be allowed to keep a particular URL around forever, and a service following Postel's law should keep honouring requests for that URL. As the RESTful Web Services Cookbook explains, if you move the resource, you should at least return a 301 (Moved Permanently) "or, in rare cases," issue a 410 (Gone) response.

2015-04-12 11:35 UTC
Sandro Magi

As Dan said, you can use an HMAC to protect certain parameters in a URL from tampering. I actually formalized this approach in a .NET library I call Clavis, and which I first discussed in detail here.

Clavis just provides a simple framework for declaring a resources parameter interface. Given that interface, Clavis provides a way to generate URLs when given a valid set of parameters, and also provides a way to safely parse values given a constructed URL. Arbitrary types can be specified as resource parameters, and there's very little boilerplate to convert to/from strings.

Parameters are protected from tampering by default, but you can declare that some are unprotected, meaning the client can change them without triggering a server validation error. You want this in some cases, for instance to support GET forms, like a search.

2015-04-12 12:16 UTC

Late to the party, but I wonder how you will create a hypermedia form that generates http://foo.ploeh.dk/598CB0CAC30646E1BB768596BFE91F2C rather than http://foo.ploeh.dk/products/2345.

In the case of hyperlinks, I fully agree: linking to a product should not require an specifically structured identifier. However, hypermedia is more than links alone: what if the client wants to navigate to product with a given tracking number, or a list of people with a given first name?

“Hackable URIs” are not a necessity for REST, but they are a potential/likely consequence of hypermedia forms. Such forms can be described with less expressive power if only simple string replacements are required instead of more complex transformations.

2016-02-25 11:20 UTC

Ruben, thank you for writing. This is a commonly occuring requirement, particularly when searching for particular resources, and I usually solve it with URI templates, as outlined in the RESTful Web Services Cookbook. Essentially, it involves providing URI templates like http://foo.ploeh.dk/0D89C19343DA4ED985A492DA1A8CDC53/{term}, and making sure that each template is served like a link, with a particular relationship type. For example:

<home xmlns="http://fnaah.ploeh.dk/productcatalog/2013/05">
  <link-template href="http://foo.ploeh.dk/0D89C19343DA4ED985A492DA1A8CDC53/{term}"
                 rel="http://catalog.api.ploeh.dk/docs/rels/product/search" />
</home>

It isn't perfect, but the best I've been able to come up with, and it works fairly well in practice.

2016-03-02 19:25 UTC

Sure, URI templates are a means for creating hypermedia controls. But the point is that what they generate are hackable URIs.

You still have a (semi-)hackable URI with http://foo.ploeh.dk/0D89C19343DA4ED985A492DA1A8CDC53/{term}. Even though the /0D89C19343DA4ED985A492DA1A8CDC53/ part is non-manipulable, terms can still be changed at the end. This shows that hackable URIs cannot entirely be avoided if we want to keep hypermedia controls simple. What else are hackable URIs but underspecified, out-of-band hypermedia controls?

An in-band IRI template makes the message follow the REST style by putting the control inside of the message, but it results in a hackable IRI. So, apparently, an HTTP REST API without hackable URIs is hard when there's more than just links.

2016-03-02 19:45 UTC

Ruben, it's not that you can't avoid hackable URLs. URI templates are an optimisation that you can opt to apply, but you don't have to use them. As with so many other architecture decisions, it's a trade-off.

When considering RESTful API design, it often helps to ask yourself: how would a web site running in a browser work? You don't have to copy web site mechanics one-to-one (for one, browsers utilise only GET and POST), but you can often get inspiration.

A feature like search is, in my experience, one of the clearest cases for URI templates, so let's consider other designs. How does search work on a web site? It works by asking the user to type the search term into a form, and then submitting this form to the server.

You can do the same with a RESTful design. When a client wishes to search, it'll have to perform an HTTP POST request with a well-formed request body containing the search term(s):

POST /108B618F06644379879B9F2FEA1CAE92 HTTP/1.1
Content-Type: application/json
Accept: application/json

{ "term": "chocolate" }

The response can either contain the search hits, or a link to a resource containing the hits.

Both options have potential drawbacks. If you return the search results in the response to the POST request, the client immediately receives the results, but you've lost the ability to cache them.

HTTP/1.1 200 OK
Content-Type: application/json

{
  "term": "chocolate",
  "results": [{
    "name": "Valrhona",
    "link": {
      "href": "http://example.com/B69AD9193F6E472AB75D8EBD97331E22",
      "rel": "product"
  }
  }, {
    "name": "Friis-Holm",
    "link": {
      "href": "http://example.com/3C4BEC72BFB54E1B8FADB16CCB3E7A67",
      "rel": "product"
    }
  }]
}

On the other hand, if the POST response only replies with a link to the search result, the search results themselves are cacheable, but now the client has to perform an extra request.

HTTP/1.1 303 See Other
Location: http://example.com/0D25692DE79B4AF9A73F128554A9119E

In both cases, though, you'll need to document the valid representations that a client can POST to the search resource, which means that you need to consider backwards compatibility if you want to change things, and so on. In other words, you can 'protect' the URL schema of the search resource, but not the payload schema.

If you're already exposing an API over HTTPS, caching is of marginal relevance anyway, so you may decide to use the first POST option outlined above. Sometimes, however, performance is more important than transport security. As an example, I once wrote (for a client) a publicly available music catalogue API. This API isn't protected, because the data isn't secret, and it has to be fast. Resources are cached for 6 hours, so if you're searching for something popular (say, Rihanna), odds are that you're going to get a response from an intermediary cache. URI templates work well in this scenario, but it's a deliberate performance optimisation.

2016-03-10 07:34 UTC

One can always avoid hackable URLs with trickery, but would you say that leads to a cleaner solution? I agree that search is “one of the clearest cases for URI templates”, so why bother putting in an unneeded POST request just to avoid hackable URIs? I fail to see why this would be a better REST design, given that you loose the beneficial properties of the uniform interface constraints by overloading the POST method for what is essentially a read-only operation. The trade-off here is entirely non-technical: you trade the performance of the interface for the aesthetics of its URLs.

Here's how a website would do it: it would have an HTML form for a GET request, which would naturally lead to a hackable URL. Because that's how HTML forms work: like URI templates, they result in hackable URLs. It's unavoidable. Hackable URLs are a direct consequence of common hypermedia controls, which are a necessity to realize the hypermedia constraint. You can swim against the current and make more difficult hypermedia controls, but to whose benefit? It's much easier to just accept that hackability is a side-effect of HTML forms and URI templates.

Therefore, your thesis that one should avoid hackable URIs breaks down at this point. APIs with non-hackable URIs are inherently incompatible with URI templates—and why throw away this excellent means of affordance? After all, the most straightforward way to realize search functionality is with an in-band URI template. And URI templates lead to hackable URIs. Hence, hackable URIs occur in REST APIs. You can work around this, but it's quite artificial to do so. In essence, you're using the POST request to obfuscate the URI; having such a translation step for purely aesthetic motives complicates the API and sacrifices caching for no good reason. Hackable URIs are definitely the lesser evil of the two.

So, would you avoid hackable URIs at any cost? Or can you agree that they indeed have a place if we like to keep using the hypermedia controls that have been standardized?

2016-03-10 23:50 UTC

Ruben, my intent with this post was to highlight a problem I've experienced when publishing REST APIs, and that I've attempted to explain in the section titled Keep clients safe. I wish that problem didn't exist, but in my experience, it does. It most likely depends on a lot of factors, so if you don't have that problem, you can disregard the entire article.

The issue isn't that I want to make URLs opaque. The real issue at stake is that I want clients to follow links, because it gives me options to evolve the API. Consider the example in this article. If a client wants to see a user's messages, it should follow links. If, however, a client developer writes the code in such a way that it'll explicitly look for a root > users > user details > user messages route, little is gained.

A client looking for user messages should be implemented with a prioritised rule set:

  1. Look for a user-messages link. If found, follow it. Done.
  2. Look for a user-details link. If found, follow it, and apply this rule list again from the top.
  3. Look for a user link. If found, follow it, and apply this rule list again from the top.
Actually, it's more complex than outlined here, because the rule set may have to dynamically change based on the context, but I hope it gets the main point across.

If client developers can guess, and thereby 'hack', my service's URLs, I can't restructure my API. It's not really a question of aesthetics; it's not that I want to be able to rename URLs. The real issue is that sometimes, when developing software, you discover that you've modelled a concept the wrong way. Perhaps you've conflated two concepts that ought to be treated as separate resources. Perhaps you've create a false distinction, but now you realise that two concepts that you thought were separate, is really the same. Therefore, sometimes you want to split one type of resource into two, or more, or you want to combine two or more different types of resources into one.

If clients follow links, you have a chance to pull off such radical restructuring. If clients have hard-coded URLs, you're out of luck.

2016-03-12 16:28 UTC

Thank you for an interesting post and discussion. I am no expert wrt REST but I have a comment (posed as a question) and a question.

If client developers abuse transparent URLs (that are provided in a level 3 api) isn't the fact that the client stops working when the URLs change their problem? And sure, they can guess other URLs, just like in the browser we can guess other pages. Doesn't mean the application stage should let them access those pages etc.

And now the question. Are there any libraries or tools that allow REST client applications to easily get a list of the links and "follow the links" (i.e. do an API call)? I can see a client having slots for possible / expected links, filling those that exist, and calling them. Is there any library to help this?

2016-06-23 15:18 UTC

Ashley, thank you for writing. In the end, as you point out, it's the client's problem if they don't adhere to the contract. In some scenarios, that's the end of the story. In other scenarios (e.g. the ones I often find myself in), there's a major client that has a lot of clout, and if those developers find that you've moved their cheese, their default reaction is that it's your fault - not theirs. In such cases, it can be political to ensure that there's no doubt about the contract.

I'm not aware of any libraries apart from HttpClient. It doesn't do any of the things you ask about, because it's too low-level, but at least it enables you to build a sensible client API on top of it.

2016-06-23 21:05 UTC

Herding Code podcast

Wednesday, 01 May 2013 10:26:00 UTC

I'm on Herding Code.

At the Danish Developer Conference 2013 I had the pleasure of meeting Jon Galloway, who interviewed me for a Herding Code podcast. In this interview, we talk about AutoFixture, testing (or not testing) trivial code, as well as lots of the unit testing topics I also cover in my Pluralsight course on Advanced Unit Testing.


REST lesson learned: Avoid 204 responses

Tuesday, 30 April 2013 10:46:00 UTC

Avoid 204 responses if you're building a HATEOAS application.

This is a lesson about REST API design that I learned while building non-trivial REST APIs. In order to be as supportive of the client as possible, a REST API should not return 204 (No Content) responses.

From the service's perspective, a 204 (No Content) response may be a perfectly valid response to a POST, PUT or DELETE request. Particularly, for a DELETE request it seems very appropriate, because what else can you say?

However, from the perspective of a proper HATEOAS-aware client, a 204 response is problematic because there are no links to follow. When hypermedia acts as the engine of application state, when there are no links, there's no state. In other words, a 204 response throws away all application state.

If a client encounters a 204 response, it can either give up, go to the entry point of the API, or go back to the previous resource it visited. Neither option is particularly good.

Giving up is not a good option if there's still work to do. Essentially, this is equivalent to a crashing client.

Going to the entry point of the API may allow the client to move on, doing what it was doing, but state may still be lost.

Going back (the equivalent of using your browser's back button) may be the best option, but has a couple of problems: First, if the client just did a DELETE, the previous resource in the history may now be gone (it was just deleted). The client would have to go back twice to arrive at a proper resource. Second, while your browser has built-in history, a programmatic HTTP client probably hasn't. You could add that feature to your client, but it would require more work. Once more, it would require the client to maintain state, which means that you'd be moving state from hypermedia to the client. It's just not a HATEOAS-compliant approach.

A good REST API should make it easy to be a client. While this is only a variation of Postel's law, I also like to think of this in terms or courtesy. The service has a lot of information available to it, so it might as well be courteous and help the client by sharing appropriate pieces of information.

Instead of a 204 (No Content) response, tell the client what it can do now.

Responding to POST requests #

An HTTP POST request often represents some sort of Command - that is: an intent to produce side effects. If the service handles the request synchronously, it should return the result of invoking the Command.

A common POST action is to create a new resource. At the very least, the REST API should return a 201 (Created) with a proper Location header. That's all described in detail in the RESTful Web Services Cookbook.

Even if the API decides to handle the request asynchronously, it can provide one or more links. Actually, the specification of 202 (Accepted) says that the "entity returned with this response SHOULD include an indication of the request's current status and [...] a pointer to a status monitor". That sounds like a link to me.

Responding to PUT requests #

An HTTP PUT request is often intended to update the state of a particular resource. Instead of returning 204 (No Content), the API should be courteous and return the new state of the resource. You may think that this is redundant because the client just transferred the state of the resource to the service, so why pay transmission costs to get the resource back from the service?

Even if a client PUTs the entire state of a resource to the API, the API may still have information about the resource not available to the client. The most universal example is probably the resource's ETag. If the client wishes to do further processing on the resource, it's likely going to need the ETag value sooner or later.

The representation of the resource may also include optional or denormalized data that the client may not have. If that data is optional, the client can make a PUT without it, but it might want that data to display to a user, so it would have to read the latest state of the resource regardless.

Responding to DELETE requests #

A DELETE request represents the intent to delete a resource. Thus, if the service successfully handles a DELETE request, what else can it do than returning a 204 (No Content)? After all, the resource has just been removed.

A resource is often a member of a collection, or otherwise 'owned' by a container. As an example, http://foo.ploeh.dk/api/tags/rock represents a "rock" tag, but another way of looking at it is that the /rock resource is contained within the tags container (which is itself a resource). This should be familiar to Atom Pub users.

Imagine that you want to delete the http://foo.ploeh.dk/api/tags/rock resource. In order to accomplish that goal, you issue a DELETE request against it. If all your client gets back is a 204 (No Content), it's just lost its context. Where does it go from there? Unless you keep state on the client, you don't know where you came from.

Instead of returning 204 (No Content), the API should be helpful and suggest places to go. In this example I think one obvious link to provide is to http://foo.ploeh.dk/api/tags - the container from which the client just deleted a resource. Perhaps the client wishes to delete more resources, so that would be a helpful link.

Responding to GET requests #

Returning 204 (No Content) as a response to a GET request is a bit weird, but probably not completely unheard of. Such a resource would represent a flag or semaphore, where the existence of the resource (as opposed to a 404 (Not Found)) signifies something. However, consider being helpful to the client and at least provide a link it can use to move on.

Summary #

Technically, responding with 204 (No Content) is perfectly valid, but if a REST API does that, its clients lose their current context. A service should be helpful and inform clients where they can go from the current resource. This will make client development easier, and only puts a small burden on the service.


Comments

I'm going to have to disagree with your interpretation of HTTP 204 NO CONTENT and the DELETE method. The HTTP/1.1 specification clearly states The server has fulfilled the request but does not need to return an entity-body, and might want to return updated metainformation. The response MAY include new or updated metainformation in the form of entity-headers, which if present SHOULD be associated with the requested variant. So you can still have a response body without entity information but contain meta information maintaining HATEOS.
2015-04-02 17:24 UTC

Xander, thank you for writing. If you read this article carefully, you may notice that it's not an exegesis of the HTTP specification, but a guideline to writing pleasant and useful RESTful services.

The bottom line is that if you're a client developer trying to follow links, 204 responses are annoying, which isn't the same as to say that they're invalid (they're not).

Could the service put links in the headers? Perhaps it could, but then it'd be forcing the client developer to do more work (by looking after links in two places).

Forcing client developers to do more work scales poorly (in terms of effort, not performance). A service developer needs to do a particular piece of work once, after which it's available to all client developers (even if there are hundreds of clients).

If a service doesn't provide a consistent API, every client developer has to duplicate the work the service developer could have done once. Every time that happens, you risk introducing bugs (on the client side). The end result is a more brittle system.

In the end, this has nothing to do with HTTP, but with Postel's law.

2015-04-02 18:35 UTC

Mark, I understand why you might say that about link headers, but not all media types are hypermedia (image/png, audio/mp4, video/ogg to name a few) whereas all HTTP header sections are. Given that, wouldn't it be wise for a client to always maintain awareness of protocol-level linking and, when a media type offers links, maintain awareness of that as well?

Since you mentioned client/server development, it's worth considering that once a given domain (be it 'people', 'autos', 'nutrition', etc.) has become cemented with standard media types and link relations, there would probably be a community effort to develop client and server libraries and applications for varying languages and platforms so that each development team doesn't have to write and test their own implementations from scratch (see: HTML browsers and frameworks.)

It makes sense that the 'World Wide Web' has become standard and RESTful; everyone is interested in the domain of general-purpose knowledge and entertainment. Unfortunately I think it will still be a while before we see many other, more specific domains become standardized to that same degree.

2015-06-12 13:50 UTC

Derek, thank you for writing. It's a good point that some media types like images, audio, etc. don't have a natural place to put meta-data like links. Conceivably, that might even include vendor-specific media types where, for some reason or other, links don't fit.

In such cases, I think that links in headers sounds preferable to no links at all. In the presence of such media types, my argument against links in headers no longer holds.

Once more, we've learned that no advice is independent of context. In the context of a RESTful API where such media types may occur, clients would have to be aware that links can appear in headers as well as in the body of any document. In such APIs, 204 responses may be perfectly fine. Here, the extra work of looking two places (both in the body, and in headers) is warranted.

In contexts of APIs where such media types don't occur, clients have no particular reason to expect links in headers. In this case, I still maintain that 204 responses put an unwarranted burden on client developers.

So far, I've only been involved with APIs without such special media types, so my initial advice was obviously coloured by my experience. Thus, I appreciate the alternative perspective.

2015-06-14 14:22 UTC

REST lesson learned: Avoid user-supplied data in URI segments

Monday, 29 April 2013 12:07:00 UTC

Be careful with user-supplied data in URI segments.

This is a lesson about design of REST APIs that I learned the hard way: if you built service URIs from dynamic data, be careful with the data you allow in URI segments.

Some HTTP frameworks (e.g. the ASP.NET Web API) let you define URI templates or routes that handle incoming requests - e.g.

routeTemplate: "api/{controller}/{id}"

which means that a request to http://foo.ploeh.dk/api/fnaah/sgryt would map the value "fnaah" to controller, and the value "sgryt" to id.

If you are building a level 2 or less service, you may even publish your URI templates to consumers. If you are building a level 3 RESTful API, you may not be publishing your URI templates, but you can still use them internally.

Avoid user-supplied data in URI segments #

Be careful with the source of data you allow to populate URI segments of your URI template. At one point, I was involved with designing a REST API that (among other things) included a 'tag cloud' feature. If you wanted to see the contents of a specific tag, you could navigate to it.

Tags were all user-defined strings, and they had no internal ID, so our first attempt was to simply treat the value of the tag as the ID. That seemed reasonable, because we wanted the tag resource to list all the resources with that particular tag value. Thus, we modeled the URI template for tag resources like the above route template.

That worked well for a URL like http://foo.ploeh.dk/api/tags/rock because it would simply match the value "rock" to the id variable, and we could then list all resources tagged with "rock".

However, some user had defined a tag with the value of "sticky & sweet" (notice the ampersand character), which meant that when you wanted to see all resources with this link, you would have to navigate to http://foo.ploeh.dk/api/tags/sticky & sweet. However, that sort of URL is considered dangerous, and IIS will (by default) refuse to handle it.

Can you get around this by URL encoding the value? No, it's part of the request path, not part of any query string, so that's not going to work. The issue isn't that the URL is invalid, but that the server considers it to be dangerous. Even if you URL encode it, the server will decode it before handling it, and that would you leave you at square one.

You can either change the URI template so that the URL instead becomes http://foo.ploeh.dk/api/tags?id=sticky%20%26%20sweet. This URL encodes the query string (the part of the URL that comes after the ?), but gives you an ugly URL.

Another alternative is to be very strict about input validation, and only allow users to create values that are safe when used as URI segments. However, that's putting an unreasonable technical limitation on an application feature. If a user wants to tag a resource with "sticky & sweet", the service should allow it.

In the end, we used a third alternative: we assigned an internal ID to all tags and mapped back and forth so that the URL for the "sticky & sweet" tag became http://foo.ploeh.dk/api/tags/1234. Yes: that makes it impossible to guess the URL, but we were building a level 3 RESTful API, so clients are expected to follow links - not guess the URL.


REST lessons learned

Monday, 29 April 2013 00:00:00 UTC

This post provides an overview of some lessons I learned while bulding non-trivial REST APIs.

Last year I spent a good deal of the year designing and implementing a handful of non-trivial REST APIs for a customer of mine. During that process, I learned some small lessons about the design of RESTful systems that I haven't seen described elsewhere, and I want to share these lessons with you.

In order to learn the concepts and philosphy behind REST, I think that REST in Practice is a great resource (pun intended), but when it comes to practical guidance, I find the RESTful Web Services Cookbook invaluable. It's full of useful and concrete tips and tricks for building RESTful APIs, but I don't remember reading about the following lessons, that I had to learn the hard way.

There's so much hype and misrepresentation about REST that I have to point out that when I'm talking about REST, I mean full-on, level 3 REST, with resources, verbs, hypermedia controls and the works.

Each of these lessons deserves a small article of its own, but here's an overview:

I hope you find these tips useful.


How to change the Generate Property Stub refactoring code snippet in Visual Studio 2012

Tuesday, 23 April 2013 14:00:00 UTC

This post describes how to change the code template for the "Generate Property Stub" refactoring in Visual Studio 2012.

This is mostly a quick note for my own benefit (since I just spent half an hour chasing this down), but I'm posting it here on the blog, in case someone else might find it useful. Here's how to change the code snippet for the "Generate Property Stub" smart tag in Visual Studio 2012:

The template for this code is defined in

C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC#\Snippets\1033
\Refactoring\GenerateProperty - Auto Property.snippet

You can open this file and hand-edit it, but contrary to previous versions of Visual Studio, it seems that you have to restart Visual Studio before the change takes effect.

If you do this, be sure that you know what you're doing; you'd be mucking around with the internals of your Visual Studio installation.


Advanced Unit Testing Pluralsight course

Monday, 15 April 2013 18:29:00 UTC

Service announcement: my new Pluralsight course Advanced Unit Testing is now available. Read more in Pluralsight's announcement, or go straight to the course.

If you don't already have a Pluralsight account, you can get a free trial of up to 200 minutes.


FizzBuzz kata in Clojure

Thursday, 11 April 2013 06:20:00 UTC

This post describes my first experience with doing the FizzBuzz kata in Clojure.

After having looked at Clojure for some time, I finally had a bit of time to play with it, so I decided to do the FizzBuzz kata in Clojure.

Single fizzbuzz function #

Clojure has a built-in testing framework, so I wrote a Parameterized Test for a single fizzbuzz function. Using TDD, I added test cases a little at a time, attempting to follow the Transformation Priority Premise, but the end result is this:

(ns fizzbuzz.core-test
  (:use clojure.test
        fizzbuzz.core))

(deftest fizzbuzz-test
  (are [expected i] (= expected (fizzbuzz i))
       "1" 1
       "2" 2
       "Fizz" 3
       "4" 4
       "Buzz" 5
       "Fizz" 6
       "7" 7
       "8" 8
       "Fizz" 9
       "Buzz" 10
       "11" 11
       "Fizz" 12
       "13" 13
       "14" 14
       "FizzBuzz" 15
       "FizzBuzz" 30))

Clojure syntax is somewhat backwards from what I'm normally used to, but the are macro expands into a collection of tests that each evaluate whether the result of invoking the fizzbuzz function with i is equal to expected.

Through a series of transformations of the SUT, I ended up with this implementation of the fizzbuzz function:

(ns fizzbuzz.core)

(defn fizzbuzz [i]
  (cond
    (= 0 (mod i 15)) "FizzBuzz"
    (= 0 (mod i 3)) "Fizz"
    (= 0 (mod i 5)) "Buzz"
    :else (str i)))

This defines a function called fizzbuzz taking a single argument i. The cond macro evaluates each test and returns the expression associated with the first test that evauluates to true. The first test checks if i is divisible with 15 and returns "FizzBuzz" if this is the case; the next test checks if i is divisible with 3 and returns "Fizz" if this is true, and so on.

Printing a range of fizzbuzz values #

The task defined by the kata is to print all FizzBuzz values from 1 to 100, so the above function is't the final solution. The next step I took was to write a test that defines a version of the fizzbuzz function taking two parameters:

(def acceptance-expected
"1
2
Fizz
4
Buzz
Fizz
7
8
Fizz
Buzz
11
Fizz
13
14
FizzBuzz
16
17
Fizz
19
Buzz
Fizz
22
23
Fizz
Buzz
26
Fizz
28
29
FizzBuzz
31
32
Fizz
34
Buzz
Fizz
37
38
Fizz
Buzz
41
Fizz
43
44
FizzBuzz
46
47
Fizz
49
Buzz
Fizz
52
53
Fizz
Buzz
56
Fizz
58
59
FizzBuzz
61
62
Fizz
64
Buzz
Fizz
67
68
Fizz
Buzz
71
Fizz
73
74
FizzBuzz
76
77
Fizz
79
Buzz
Fizz
82
83
Fizz
Buzz
86
Fizz
88
89
FizzBuzz
91
92
Fizz
94
Buzz
Fizz
97
98
Fizz
Buzz")

(deftest acceptance-test
  (is (= acceptance-expected (fizzbuzz 1 101))))

I decided to define the acceptance-expected value outside of the test case itself, as I thought that made the test a bit more readable. The test case is defined by the is macro and states that the expected value is acceptance-expected and the actual value is the result of invoking the fizzbuzz function with two arguments: 1 as the (inclusive) start value, and 101 as the (exclusive) end value. The above code listing of fizzbuzz only accept one argument, but the new test case requires two arguments, so I added an overload to the function:

(defn fizzbuzz
  ([i]
    (cond
      (= 0 (mod i 15)) "FizzBuzz"
      (= 0 (mod i 3)) "Fizz"
      (= 0 (mod i 5)) "Buzz"
      :else (str i)))
  ([start end]
    (apply str
      (interpose "\n"
        (map fizzbuzz (range start end))))))

The previous implementation is still there, now contained within the overload taking a single argument i, but now there's also a new overload taking two arguments: start and end.

This overload generates a sequence of integers from start to end using the range function. It then maps that sequence of integers into a sequence of strings by mapping each integer to a string with the fizzbuzz function. That gives you a sequence of strings such as ("1" "2" "Fizz" "4" "Buzz").

In order to print all the FizzBuzz strings, I need to interpose a newline character between each string, which produces a new sequence of strings such as ("1" "\n" "2" "\n" "Fizz" "\n" "4" "\n" "Buzz"). To concatenate all these strings, I apply the str function to the sequence.

Printing FizzBuzz values from 1 to 100 #

The requirements of the kata is to print all FizzBuzz values from 1 to 100, and the code already does this. However, I interpret the kata as requiring a single function that takes no parameters, so I added an acceptance test case:

(deftest acceptance-test
  (is (= acceptance-expected (fizzbuzz 1 101)))
  (is (= acceptance-expected (fizzbuzz))))

Notice the second test case in the last line of code that invokes the fizzbuzz function without any parameters. It's easily resolved by adding a third overload:

(defn fizzbuzz
  ([]
    (fizzbuzz 1 101))
  ([i]
    (cond
      (= 0 (mod i 15)) "FizzBuzz"
      (= 0 (mod i 3)) "Fizz"
      (= 0 (mod i 5)) "Buzz"
      :else (str i)))
  ([start end]
    (apply str
      (interpose "\n"
        (map fizzbuzz (range start end))))))

As you can see, the first overload takes no parameters and simply invokes the previously desribed overload with the start and end arguments.

FWIW, this entire solution is structurally similar to my implementation of FizzBuzz in F#.


Ploeh blog syndication feed addresses

Tuesday, 09 April 2013 07:53:00 UTC

Service announcement about syndication feed addresses for ploeh blog.

Now that Google Reader is closing down and a lot of my readers may want to migrate their ploeh blog subscription to another service, I think it's the right time for a quick post about ploeh blog's syndication feed addresses.

When I migrated my blog to Jekyll I made it a priority to ensure a certain level of backwards compatibility. The permalinks for the old posts are still served, although they are now redirects. For that reason, I also left the old syndication feed addresses in place. Specifically, I left this old RSS feed address in place: http://blog.ploeh.dk/SyndicationService.asmx/GetRss. This feed address sort of works, but has issues.

There are several problems with the 'legacy' feed address:

  • Due to the way Jekyll works, the address actually points to an index.html file. Since a Jekyll-powered site is a static site, I don't control the server, and thus I can't manipulate the HTTP headers for individual resources. HTML files are served with the "text/html" Content-Type, which doesn't fit the XML content. Some clients seem to dislike this.
  • Google Reader has created some 'ghost' entries based on the feed. I wonder if it has anything to do with the faulty Content-Type.
  • Other users have reported that their clients (e.g. Outlook) don't like the feed. Again, I suspect it's because of the faulty Content-Type. While I think those clients should follow Postel's law, they apparently don't.
  • The address looks very implementation-specific.

However, ploeh blog has new, 'proper' syndication feed addresses, so if you're already in the process of migrating your subscriptions away from Google Reader, please take a moment to update your subscription. That 'legacy' RSS address may not stick around forever.

The proper syndication feed addresses for ploeh blog are:

Both serve responses with the "text/xml" Content-Type.


Page 22 of 44

"Our team wholeheartedly endorses Mark. His expert service provides tremendous value."
Hire me!