Instead of relying on entity IDs, use hypermedia to identify resources.

Whenever I've helped teams design HTTP APIs, sooner or later one request comes up - typically from client developers: Please add the entity ID to the representation.

In this article I'll show an alternative, but first: the normal state of affairs.

Business as usual #

It's such a common requirement that, despite admonitions not to expose IDs, I did it myself in the code base that accompanies my book Code That Fits in Your Head. This code base is a level 3 REST API, and still, I'd included the ID in the JSON representation of a reservation:

{
  "id""bf4e84130dac451b9c94049da8ea8c17",
  "at""2021-12-08T20:30:00.0000000",
  "email""snomob@example.com",
  "name""Snow Moe Beal",
  "quantity": 1
}

At least the ID is a GUID, so I'm not exposing internal database IDs.

After having written the book, the id property kept nagging me, and I wondered if it'd be possible to get rid of it. After all, in a true REST API, clients aren't supposed to construct URLs from templates. They're supposed to follow links. So why do you need the ID?

Following links #

Early on in the system's lifetime, I began signing all URLs to prevent clients from retro-engineering URLs. This also meant that most of my self-hosted integration tests were already following links:

[Theory]
[InlineData(867, 19, 10, "adur@example.net""Adrienne Ursa", 2)]
[InlineData(901, 18, 55, "emol@example.gov""Emma Olsen", 5)]
public async Task ReadSuccessfulReservation(
    int days,
    int hours,
    int minutes,
    string email,
    string name,
    int quantity)
{
    using var api = new LegacyApi();
    var at = DateTime.Today.AddDays(days).At(hours, minutes)
        .ToIso8601DateTimeString();
    var expected = Create.ReservationDto(at, email, name, quantity);
    var postResp = await api.PostReservation(expected);
    Uri address = FindReservationAddress(postResp);
 
    var getResp = await api.CreateClient().GetAsync(address);
 
    getResp.EnsureSuccessStatusCode();
    var actual = await getResp.ParseJsonContent<ReservationDto>();
    Assert.Equal(expected, actual, new ReservationDtoComparer());
    AssertUrlFormatIsIdiomatic(address);
}

This parametrised test uses xUnit.net 2.4.1 to first post a new reservation to the system, and then following the link provided in the response's Location header to verify that this resource contains a representation compatible with the reservation that was posted.

A corresponding plaintext HTTP session would start like this:

POST /restaurants/90125/reservations?sig=aco7VV%2Bh5sA3RBtrN8zI8Y9kLKGC60Gm3SioZGosXVE%3D HTTP/1.1
Content-Type: application/json
{
  "at""2021-12-08 20:30",
  "email""snomob@example.com",
  "name""Snow Moe Beal",
  "quantity": 1
}

HTTP/1.1 201 Created
Content-Type: application/json; charset=utf-8
Location: example.com/restaurants/90125/reservations/bf4e84130dac451b9c94049da8ea8c17?sig=ZVM%2[...]
{
  "id""bf4e84130dac451b9c94049da8ea8c17",
  "at""2021-12-08T20:30:00.0000000",
  "email""snomob@example.com",
  "name""Snow Moe Beal",
  "quantity": 1
}

That's the first request and response. Clients can now examine the response's headers to find the Location header. That URL is the actual, external ID of the resource, not the id property in the JSON representation.

The client can save that URL and request it whenever it needs the reservation:

GET /restaurants/90125/reservations/bf4e84130dac451b9c94049da8ea8c17?sig=ZVM%2[...] HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
{
  "id""bf4e84130dac451b9c94049da8ea8c17",
  "at""2021-12-08T20:30:00.0000000",
  "email""snomob@example.com",
  "name""Snow Moe Beal",
  "quantity": 1
}

The actual, intended use of the API doesn't rely on the id property, neither do the tests.

Based on this consistent design principle, I had reason to hope that I'd be able to remove the id property.

Breaking change #

My motivation for making this change was to educate myself. I wanted to see if it would be possible to design a REST API that doesn't expose IDs in their JSON (or XML) representations. Usually I'm having trouble doing this in practice because when I'm consulting, I'm typically present to help the organisation with test-driven development and how to organise their code. It's always hard to learn new ways of doing things, and I don't wish to overwhelm my clients with too many changes all at once.

So I usually let them do level 2 APIs because that's what they're comfortable with. With that style of HTTP API design, it's hard to avoid id fields.

This wasn't a constraint for the book's code, so I'd gone full REST on that API, and I'm happy that I did. By habit, though, I'd exposed the id property in JSON, and I now wanted to perform an experiment: Could I remove the field?

A word of warning: You can't just remove a JSON property from a production API. That would constitute a breaking change, and even though clients aren't supposed to use the id, Hyrum's law says that someone somewhere probably already is.

This is just an experiment that I carried out on a separate Git branch, for my own edification.

Leaning on the compiler #

As outlined, I had relatively strong faith in my test suite, so I decided to modify the Data Transfer Object (DTO) in question. Before the change, it looked like this:

public sealed class ReservationDto
{
    public LinkDto[]? Links { getset; }
    public string? Id { getset; }
    public string? At { getset; }
    public string? Email { getset; }
    public string? Name { getset; }
    public int Quantity { getset; }
}

At first, I simply tried to delete the Id property, but while it turned out to be not too bad in general, it did break one feature: The ability of the LinksFilter to generate links to reservations. Instead, I changed the Id property to be internal:

public sealed class ReservationDto
{
    public LinkDto[]? Links { getset; }
    internal string? Id { getset; }
    public string? At { getset; }
    public string? Email { getset; }
    public string? Name { getset; }
    public int Quantity { getset; }
}

This enables the LinksFilter and other internal code to still access the Id property, while the unit tests no longer could. As expected, this change caused some compiler errors. That was expected, and my plan was to lean on the compiler, as Michael Feathers describes in Working Effectively with Legacy Code.

As I had hoped, relatively few things broke, and they were fixed in 5-10 minutes. Once everything compiled, I ran the tests. Only a single test failed, and this was a unit test that used some Back Door Manipulation, as xUnit Test Patterns terms it. I'll return to that test in a future article.

None of my self-hosted integration tests failed.

ID-free interaction #

Since clients are supposed to follow links, they can still do so. For example, a maître d'hôtel might request the day's schedule:

GET /restaurants/90125/schedule/2021/12/8?sig=82fosBYsE9zSKkA4Biy5t%2BFMxl71XiLlFKaI2E[...] HTTP/1.1
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyZXN0YXVyYW50IjpbIjEiLCIyMTEyIiwi[...]

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
{
  "name""The Vatican Cellar",
  "year": 2021,
  "month": 12,
  "day": 8,
  "days": [
    {
      "date""2021-12-08",
      "entries": [
        {
          "time""20:30:00",
          "reservations": [
            {
              "links": [
                {
                  "rel""urn:reservation",
                  "href""http://example.com/restaurants/90125/reservations/bf4e84130dac4[...]"
                }
              ],
              "at""2021-12-08T20:30:00.0000000",
              "email""snomob@example.com",
              "name""Snow Moe Beal",
              "quantity": 1
            }
          ]
        }
      ]
    }
  ]
}

I've edited the response quite heavily by removing other links, and so on.

Clients that wish to navigate to Snow Moe Beal's reservation must locate its urn:reservation link and use the corresponding href value. This is an opaque URL that clients can use to make requests:

GET /restaurants/90125/reservations/bf4e84130dac451b9c94049da8ea8c17?sig=vxkBT1g1GHRmx[...] HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
{
  "at""2021-12-08T20:30:00.0000000",
  "email""snomob@example.com",
  "name""Snow Moe Beal",
  "quantity": 1
}

In none of these interactions do clients rely on the id property - which is also gone now. It's gone because the Id property on the C# DTO is internal, which means that it's not being rendered.

Mission accomplished.

Conclusion #

It always grates on me when I have to add an id property to a representation in an HTTP API. It's often necessary when working with a level 2 API, but with a proper hypermedia-driven REST API, it may not be necessary.

At least, the experiment I performed with the code base from my book Code That Fits in Your Head indicates that this may be so.


Comments

It seems to me that this approach will cause problems if 3rd parties need to integrate with your API in a way where they themselves need to store references to entities in your system. For example, they may expose your entities to their users with additional data in their systems/integrations. Sure, it is possible for them to use the URI as a primary key (if you guarantee a sensible max URI length; another can of worms), but if you internally use INT or UNIQUEIDENTIFIER as your primary key, I would not want to force them to use VARCHAR(whatever) as primary key.

Therefore, in all our APIs, we document in the API specification that the IDs, though required by JSON:API (which we follow) to be passed as string values for consistency, can be safely assumed to be integers (or GUIDs, if relevant). We even document that they are 32-bit ints, so any clients know they can safely use INT fields instead of BIGINT.

JSON:API requires all entities to have a single ID. For obvious reasons, IDs should be stable. Therefore, for entities that represent an association between two other entities and do not have a separate, persisted ID, we have a need to have API IDs that contain information about the associated entities. To combat Hyrum's law, we typically concatenate the associated IDs using a known delimiter and encode the resulting string using a non-standard, URL-friendly encoding (i.e., not Base64, which may contain non-URL-friendly characters and is often obvious). This way, the IDs appear opaque to API clients. Of course, the format of these IDs are not documented in our API specifications, as they are not intended to be stored. Instead, the actual association is documented and the related entities retrievable (of course, since this information inherent to the entity's very nature), and the associated IDs may be used by clients in a multi-column primary key, just like we do.

All of the above assumes that the integrating clients use a SQL database or similar. Let's face it; many do. If you have (or may hve in the future) a single client that do this, you have to take the above into account.

2021-09-20 9:07 UTC

Christer, thank you for writing. I think that one of the problems with discussions about REST APIs, or just HTTP APIs in general, is that people use them for all sorts of things. At one extreme, you have Backends For Frontends, where, if you aren't writing the API with the single client in mind, you're doing something wrong. At the other extreme, you have APIs that may have uncountable and unknown clients. When I write about REST, I mostly have the latter kind in mind.

When designing APIs for many unknown clients, it makes little sense to take 'special needs' into account. Different clients may present mutually exclusive requirements.

Clients that need to 'bookmark' REST resources in a database can do that by defining two columns: one an ordinary primary key column on which the table defines its clustered index, and another column for the link value itself, with a UNIQUE constraint. Something like this (in T-SQL dialect):

CREATE TABLE [dbo].[Foo] (
    [Id]           INT              NOT NULL IDENTITY PRIMARY KEY CLUSTERED,
    [Address]      NVARCHAR (850)   NOT NULL UNIQUE NONCLUSTERED,
    -- Other columns go here...
);

Client code can look up an API resource on internal key, or on address, as required.

2021-09-21 16:27 UTC

Your URLs include a signature, which changes if you need to switch signing keys. Furthermore, the base URL for your API may change. The entities are still the same; the restaurant previously at old.domain/restaurants/1?sig=abc is the same as the restaurant now at new.domain/restaurants/1?sig=123. With your proposed bookmark-based solution, the API clients would effectively lose the associations in their system.

Also, indexing a very long varchar column probably works fine for tables that are fairly small and not overly busy. But for large and/or busy tables containing entities that are created every second of every day (say, passages through gates at hundreds of large construction sites, which is one of the domains I work with), I think that the performance would suffer unreasonably. (Admittedly, I have of course not measured this; this is just speculation, and anyway not my main point.)

You say you write APIs with arbitrary clients in mind. I do, too. That is one of the reasons I design my APIs at REST level 2 instead of 3. (JSON:API does offer some possibility of just "following links" if the client wishes to do that, though it is does not allow for APIs that are fully level 3/HATEOAS.) Having stable IDs with well-known formats and being able to construct URLs seems pragmatically like a good solution that keeps client developers happy. I do not have decades of experience, but I have never encountered clients who have been unhappy with my decision to go for level 2 instead of level 3. (I imagine I would have encountered some resistance in the opposite case, though that is pure speculation on my part.) Furthermore, I have never encountered the need for breaking changes that would be non-breaking by level 3 standards.

You say it makes little sense to take "special needs" into account. Idealistically, I agree. Pragmatically, 1) SQL databases are so ubiquitous and have been for such a long time that making life better for those developers by including an ID with a guaranteed format seems like a fair decision, and 2) our APIs (and many others, I assume) are created not just for 3rd party integration but also for one or more 1st party front-ends, which naturally tends to receive some preferential treatment (e.g. features and designs that probably aren't useful to other clients).

2021-09-21 20:56 UTC

Christer, thank you for writing. It's possible that I'm going about this the wrong way. I only report on what's been working for me, but that said, while I do have decades of general programming experience, I don't have decades of REST experience. I designed my first REST API in 2012.

Additionally, just because one style of API design works well, that doesn't rule out that other types of design also work.

Finally, this particular article is an experiment. I've never done something like this in the wild, so it's possible that it does have unforeseen issues.

A couple of answers to your various points, though:

I don't foresee having to change signing keys, but if that happens, it'd be a breaking change to remove support for old keys. One might have to, instead, retire old signing keys in the same way one can retire old service versions. Even if a key gets 'compromised', it's not an immediate issue. It only means that any client that possesses the leaked key can construct URLs directly by retro-engineering implied URL templates. This would still be undocumented and unsupported use of the API, which means that ultimately, it'd be against the client developers' own self-interest in doing that.

Signing the URLs isn't a security measure; it's more like a nudge.

"our APIs (and many others, I assume) are created not just for 3rd party integration but also for one or more 1st party front-ends, which naturally tends to receive some preferential treatment"

I've written APIs like that as well, and if there's one thing I've learned from doing that is that if I'm ever again put in charge of such an API, I'll strongly resist giving preferential treatment to any clients. If a particular client needs a particular feature, the client team can develop and maintain a Backend for Frontend, which bases its own implementation on the general-purpose API.

My experience with supporting particular clients is that client needs evolve much faster than APIs. This makes sense. Someone wants to do A/B testing on the client's user interface. Depending on the outcome of such a test, at least one of the supporting features will now be obsolete. I'm not much inclined having to support such features in an API where backwards compatibility is critical.

But again, these things are never clear-cut. Much depends on the overall goals of the organisation - and these may also change over time. I'm not claiming that my way is best - only that it's possible.

2021-09-23 8:07 UTC


Wish to comment?

You can add a comment to this post by sending me a pull request. Alternatively, you can discuss this post on Twitter or somewhere else with a permalink. Ping me with the link, and I may respond.

Published

Monday, 20 September 2021 06:21:00 UTC

Tags



"Our team wholeheartedly endorses Mark. His expert service provides tremendous value."
Hire me!
Published: Monday, 20 September 2021 06:21:00 UTC