Advancing Drupal's web services

In an earlier blog post, I looked at the web services solutions available in Drupal 8 and compared their strengths and weaknesses. That blog post was intended to help developers choose between different solutions when building Drupal 8 sites. In this blog post, I want to talk about how to advance Drupal's web services beyond Drupal 8.1 for the benefit of Drupal core contributors, module creators and technical decision-makers.

I believe it is really important to continue advancing Drupal's web services support. There are powerful market trends that oblige us to keep focused on this: integration with diverse systems having their own APIs, the proliferation of new devices, the expanding Internet of Things (IoT), and the widening adoption of JavaScript frameworks. All of these depend to some degree on robust web services.

Moreover, newer headless content-as-a-service solutions (e.g. Contentful, Prismic.io, Backand and CloudCMS) have entered the market and represent a widening interest in content repositories enabling more flexible content delivery. They provide content modeling tools, easy-to-use tools to construct REST APIs, and SDKs for different programming languages and client-side frameworks.

In my view, we need to do the following, which I summarize in each of the following sections: (1) facilitate a single robust REST module in core; (2) add functionality to help web services modules more easily query and manipulate Drupal's entity graph; (3) incorporate GraphQL and JSON API out of the box; and (4) add SDKs enabling easy integration with Drupal. Though I shared some of this in my DrupalCon New Orleans keynote, I wanted to provide more details in this blog post. I'm hoping to discuss this and revise it based on feedback from you.

One great REST module in core

While core REST can be enabled with only a few configuration changes, the full extent of possibilities in Drupal is only unlocked either when leveraging modules which add to or work alongside core REST's functionality, such as Services or RELAXed, or when augmenting core REST's capabilities with additional resources to interact with (by providing corresponding plugins) or using other custom code.

Having such disparate REST modules complicates the experience. These REST modules have overlapping or conflicting feature sets, which are shown in the following table.

Feature Core REST RELAXed Services Ideal core REST
Content entity CRUD Yes Yes Yes Yes
Configuration entity CRUD Create resource plugin (issue) Create resource plugin Yes Yes
Custom resources Create resource plugin Create resource plugin Create Services plugin Possible without code
Custom routes Create resource plugin or Views REST export (GET) Create resource plugin Configurable route prefixes Possible without code
Translations Not yet (issue) Yes Create Services plugin Yes
Revisions Create resource plugin Yes Create Services plugin Yes
File attachments Create resource plugin Yes Create Services plugin Yes
Authenticated user resources (log in/out, password reset) Not yet (issue) No User login and logout Yes

I would like to see a convergence where all of these can be achieved in Drupal core with minimal configuration and minimal code.

Working with Drupal's entity graph

Recently, a discussion at DrupalCon New Orleans with key contributors to the core REST modules, maintainers of important contributed web services modules, and external observers led to a proposed path forward for all of Drupal's web services.

Web services entity graph
A visual example of an entity graph in Drupal.

Buried inside Drupal is an "entity graph" over which different API approaches like traditional REST, JSON API, and GraphQL can be layered. These varied approaches all traverse and manipulate Drupal's entity graph, with differences solely in the syntax and features made possible by that syntax. Unlike core's REST API which only returns a single level (single entity or lists of entities), GraphQL and JSON API can return multiple levels of nested entities as the result of a single query. To better understand what this means, have a look at the GraphQL demo video I shared in my DrupalCon Barcelona keynote.

What we concluded at DrupalCon New Orleans is that Drupal's GraphQL and JSON API implementations require a substantial amount of custom code to traverse and manipulate Drupal's entity graph, that there was a lot of duplication in that code, and that there is an opportunity to provide more flexibility and simplicity. Therefore, it was agreed that we should first focus on building an "entity graph iterator" that can be reused by JSON API, GraphQL, and other modules.

This entity graph iterator would also enable manipulation of the graph, e.g. for aliasing fields in the graph or simplifying the structure. For example, the difference between Drupal's "base fields" and "configured fields" is irrelevant to an application developer using Drupal's web services API, but Drupal's responses leak this internal distinction by prefixing configured fields with field_ (see the left column in the table below). By the same token, all fields, even if they carry single values, expose the verbosity of Drupal's typed data system by being presented as arrays (see the left column in the table below). While there are both advantages and disadvantages to exposing single-value fields as arrays, many developers prefer more control over the output or the ability to opt into simpler outputs.

A good Drupal entity graph iterator would simplify the development of Drupal web service APIs, provide more flexibility over naming and structure, and eliminate duplicate code.

Current core REST (shortened response) Ideal core REST (shortened response)
{
  "nid": [
    {
      "value": "2"
    }
  ],
  "title": [
    {
      "value": "Lorem ipsum"
    }
  ],
  "field_product_number": [
    {
      "value": "35"
    }
  ],
  "field_image": [
    {
      "target_id": "2",
      "alt": "Image",
      "title": "Hover text",
      "width": "210",
      "height": "281",
      "url": "http://site.com/x.jpg"
    }
  ]
}
{
  "nid": "2"
  "title": "Lorem ipsum",
  "product_number": {
    "value": 35
  },
  "image": {
    "target_id": 2,
    "alt": "Image",
    "title": "Hover text",
    "width": 210,
    "height": 281,
    "url": "http://site.com/x.jpg"
  }
}

GraphQL and JSON API in core

We should acknowledge simultaneously that the wider JavaScript community is beginning to embrace different approaches, like JSON API and GraphQL, which both enable complex relational queries that require fewer requests between Drupal and the client (thanks to the ability to follow relationships, as mentioned in the section concerning the entity graph).

While both JSON API and GraphQL are preferred over traditional REST due to their ability to provide nested entity relationships, GraphQL goes a step further than JSON API by facilitating explicitly client-driven queries, in which the client dictates its data requirements.

I believe that GraphQL and JSON API in core would be a big win for those building decoupled applications with Drupal, and these modules can use existing foundations in Drupal 8 such as the Serialization module. Furthermore, Drupal's own built-in JavaScript-driven UIs could benefit tremendously from GraphQL and JSON API. I'd love to see them in core rather than as contributed modules, as we could leverage them when building decoupled applications backed by Drupal or exchanging data with other server-side implementations. We could also "eat our own dog food" by using them to power JavaScript-driven UIs for block placement, media management, and other administrative interfaces. I can even see a future where Views and GraphQL are closely integrated.

Web services rest json grapql
A comparison of different API approaches for Drupal 8, with amended and simplified payloads for illustrative purposes.

SDKs to consume web services

While a unified REST API and support for GraphQL and JSON API would dramatically improve Drupal as a web services back end, we need to be attentive to the needs of consumers of those web services as well by providing SDKs and helper libraries for developers new to Drupal.

An SDK could make it easy to retrieve an article node, modify a field, and send it back without having to learn the details of Drupal's particular REST API implementation or the structure of Drupal's underlying data storage. For example, this would allow front-end developers to not have to deal with the details of single- versus multi-value fields, optional vs required fields, validation errors, and so on. As an additional example, incorporating user account creation and password change requests into decoupled applications would empower front-end developers building these forms on a decoupled front end such that they would not need to know anything about how Drupal performs user authentication.

As starting points for JavaScript applications, native mobile applications, and even other back-end applications, these SDKs could handle authenticating against the API and juggling of the correct routes to resources without the front-end developer needing an understanding of those nuances.

In fact, at Acquia we're now in the early stages of building the first of several SDKs for consuming and manipulating data via Drupal 8's REST API. Waterwheel (previously Hydrant), a new generic helper library intended for JavaScript developers building applications backed by Drupal, is the work of Acquia's Matt Grill and Preston So, and it is already seeing community contributions. We're eager to share our work more widely and welcome new contributors.

Conclusion

I believe that it is important to have first-class web services in Drupal out of the box in order to enable top-notch APIs and continue our evolution to become API-first.

In parallel with our ongoing work on shoring up our REST module in core, we should provide the underpinnings for even richer web services solutions in the future. With reusable helper functionality that operates on Drupal's entity graph available in core, we open the door to GraphQL, JSON API, and even our current core REST implementation eventually relying on the same robust foundation. Both GraphQL and JSON API could also be promising modules in core. Last but not least, SDKs like Hydrant that empower developers to work with Drupal without learning its complexities will further advance our web services.

Collectively, these tracks of work will make Drupal uniquely compelling for application developers within our own community and well beyond.

Special thanks to Preston So for contributions to this blog post and to Moshe Weitzman, Kyle Browning, Kris Vanderwater, Wim Leers, Sebastian Siemssen, Tim Millwood, Ted Bowman, and Mateu Aguiló Bosch for their feedback during its writing.

Comments

dawehner (not verified):

This 'field_' prefix is just a vestige of the old CCK days and is a pure UI feature. You can actually even configure the prefix in the field_ui.settings.yml so it might be worth to reconsider changing this default value from 'field_' to something else like nothing.

Wim Leers (not verified):

+1

It still serves a purpose — although a limited one — to prevent collisions between a field name you configure and a pre-existing base field… or a base field that is added later.

Nate Craddock (not verified):

Great post! I'm happy to read a bit more about this initiative. Availability of our content via JSON is one of the key ways that Drupal has grown for us and this aligns with what we envision for our D8 implementation for the future. My only question is how do I get involved and help out :)

e0ipso (not verified):

One of the additional benefits of having a graph traversing library is that we can generate a schema of Drupal's data model. That schema is needed for GraphQL and can be leveraged by JSON API to determine what conforms a resource (list of attributes, relationships, field aliases, deprecated fields, …). This can be incredibly useful if you can, then, override it and tweak it to obtain the API output you want (map 'field_category' to 'categories', map 'body[value]' to 'description', define formatters to transform timestamps to ISO dates, …).

In the spirit of collaboration, it would be great to define a common way to version these web services. The goal should be to make this specification agnostic so other implementations, if any, can leverage this too.

Finally, a nit remark is that the above graphic could be further refined if the JSON API example uses: GET /jsonapi/node/1?include=field_author&fields[node]=title&fields[user]=name. In that scenario the output would only include the title attribute for the node and the author's name for the included user.

Dries:

Thanks for the suggestions. The JSON API example was purposely simplified, but your suggestion is a good one.

Ryan Szrama (not verified):

Perhaps I'm misunderstanding JSON API, but isn't it simply another media type to be used for REST responses? i.e. are REST and JSON API two different things, or are we really just talking about creating a new serialization module to replace the HAL+JSON module?

fwiw, REST in general doesn't care how you format your document or whether or not you embed resources. It's just up to the media type to identify how a client interpreting the response should locate the embedded resources.

In case it's helpful to folks preparing to implement this, I'll point to the https://www.drupal.org/project/commerce_services module as a potential example. We support embedding referenced entities to the n depth in request responses for all of our resources, as Drupal Commerce entities are often collections of references.

This would return just product display node data:

GET /product­-displays?expand_entities=0

Whereas this would return product display node data and the product entities they reference:

GET /product-displays?expand_entities=1

(And in the event that product entities also have references, you could expand to the 2nd, 3rd, etc. level.)

We also support what I called field flattening by default (can be disabled on any request), which did exactly what you propose - turns any field value array for single value, single column field types into a simple scalar. It also removes the language abstraction in the value array on single language sites.

I posted examples for all of these options in the module's PDF guide here:

https://www.drupal.org/files/issues/Commerce%20Services%20Documentation…

And I highly recommend using Apigee's Web API Design e-book as a guide! (Link in the Commerce Services project page.)

e0ipso (not verified):

Perhaps I'm misunderstanding JSON API, but isn't it simply another media type to be used for REST responses?

I would say it is. The big difference I see between HAL and JSON API is that JSON API is more explicit about the solutions to the problems that we've been having with REST in core (bloated outputs, multiple requests, list operations, …).

fwiw, REST in general doesn't care how you format your document or whether or not you embed resources.

That is true. The point of JSON API, however, is, and I quote, «JSON API can be your anti-bikeshedding tool». If we are decided to implement all those features we say we need (and some others), you can benefit from doing it in an standard way. By adopting JSON API we are gaining automatic integration with all the tools that exist out there (ef4's presentation in NOLA is a great example of that).

So I would say that JSON API will not provide any ground breaking feature, but will standarize the solution to our current problems with web services. And if we do implement a ground breaking feature we can have it inside of JSON API since it has an extension negotiation protocol.

Ryan Szrama (not verified):

Great, just making sure I understood it properly. For some reason, the JSON API website didn't seem to spell it out quite so explicitly (at least where I read). I suppose on 8.x Drupal would still need to include HAL+JSON, but would we essentially deprecate it in favor of JSON API?

Preston So (not verified):

I don't think so, because conceivably there are already many sites using HAL as their media type. Part of the mission of the team working on the type graph iterator is to build a strong foundation for the benefit of all formats moving forward, including HAL, whose module would foreseeably be refactored to accommodate this foundation. In 8.x and beyond, I would like to see HAL coexist with JSON API, with the optionality to use either one at a developer's discretion.

Joel McNamee (not verified):

Dries, if I told you there's a fully REST enabled, open source identity platform that already addresses "the proliferation of new devices, [and] the expanding Internet of Things (IoT)," REST enabling "Authenticated user resources (log in/out, password reset)," and can already integrate authentication and authorization decisions from graph databases, would you be interested? There's an amazing compliment to what we (ForgeRock and Acquia) both do. Also highlighted by our mutual call-out in the Accel TechCrunch blog (https://techcrunch.com/2016/06/19/the-next-wave-in-software-is-open-ado…).