Dries Buytaert

Of all the trends in the web community, few are spreading more rapidly than decoupled (or "headless") content management systems (CMS). The evolution from websites to more interactive web applications and the need for multi-channel publishing suggest that moving to a decoupled architecture that leverages client-side frameworks is a logical next step for Drupal. For the purposes of this blog post, the term decoupled refers to a separation between the back end and one or more front ends instead of the traditional notion of service-oriented architecture.

Decoupling your content management system enables front-end developers to fully control an application or site's rendered markup and user experience. Furthermore, the use of client-side frameworks helps developers give websites application-like behavior with smoother interactivity (there is never a need to hit refresh, because new data appears automatically), optimistic feedback (a response appears before the server has processed a user's query), and non-blocking user interfaces (the user can continue to interact with the application while portions are still loading).

Another advantage of a decoupled architecture is decoupled teams. Since front-end and back-end developers no longer need to understand the full breadth and depth of a monolithic architecture, they can work independently. With the proper preparation, decoupled teams can improve the velocity of a project.

Still, it's important to temper the hype around decoupled content management with balanced analysis. Before decoupling, you need to ask yourself if you're ready to do without functionality usually provided for free by the CMS, such as layout and display management, content previews, user interface (UI) localization, form display, accessibility, authentication, crucial security features such as XSS (cross-site scripting) and CSRF (cross-site request forgery) protection, and last but not least, performance. Many of these have to be rewritten from scratch, or can't be implemented at all, on the client-side. For many projects, building a decoupled application or site on top of a CMS will result in a crippling loss of critical functionality or skyrocketing costs to rebuild missing features.

To be clear, I'm not arguing against decoupled architectures per se. Rather, I believe that decoupling needs to be motivated through ample research. Since it is still quite early in the evolution of this architectural paradigm, we need to be conscious about how and in what scenarios we decouple. In this blog post, we examine different decoupled architectures, discuss why a fully decoupled architecture is not ideal for all use cases, and present "progressive decoupling", a better approach for which Drupal 8 is well-equipped. Today, Drupal 8 is ready to deliver pages quickly and offers developers the flexibility to enhance user experience through decoupled interactive components.

Fully decoupled is not usually the best solution

Traditionally, CMSes have focused on making websites rather than web applications, but the line between them continues to blur. For example, let's imagine we are building a site delivering content-rich curated music reviews alongside an interaction-rich ticketing interface for live shows. In the past, this ticketing interface would have been a multi-step flow, but here we aim to keep visitors on the same page as they browse and purchase ticket options.

To write and organize reviews, beyond basic content management, we need site building tools to assemble content and lay it out on a page. On the other hand, for a pleasant ticket purchase experience, we need seamless interactivity. From the end user's perspective, this means a continuous, uninterrupted experience using the application and best of all, no need to refresh the page.

Using Drupal to build a content-rich site delivering music reviews is very easy thanks to its content types and extensive editorial features. We can list, categorize, and write reviews by employing the user interfaces provided by Drupal. But because of Drupal's emphasis on server-side rendering rather than client-side rendering, Drupal alone does not yet satisfy our envisioned user experience. Meanwhile, off-the-shelf JavaScript MVC frameworks and native application frameworks are ill-suited for managing large amounts of content, due to the need to rebuild various content management and site building tools, and they can pay a performance penalty.

In a fully decoupled architecture, our losses from having to rebuild what Drupal gives us for free outweigh the wins from client-side frameworks. With a fully decoupled front end, we lose important aspects of the theme layer (which is tightly intertwined with Drupal APIs) such as theme hook suggestions, Twig templating, and, to varying degrees, the render pipeline. We lose content preview and nuances of creating, curating, and composing content such as layout management tools. We lose all of the advancements in accessibility and user experience that make Drupal 8 websites a great tool for both end users and site builders, like ARIA roles, improved system messages, and, most visibly, the fully integrated Drupal toolbar on non-administrative pages. Moreover, where there is elaborate interactivity, security vulnerabilities are easily introduced.

Progressive rendering with BigPipe

Fully decoupled architectures can also have a performance disadvantage. We want users to see the information they want as soon as possible (time to first paint) and be able to perform a desired action as soon as possible (time to interaction). A well-architected CMS will have the advantage over a client-side framework.

Much of the content we want to send for our music website is cacheable and mostly static, such as a navigation bar, recurring footer, and other unchanging components. Under a traditional page serving model, however, operations taking longer to execute can hold up simpler parts of the page in the pipeline and significantly delay the response to the client. In our music site example, this could be blocks containing songs a user listened to most in the last week and the song currently playing in a music player.

What we really need is a means of selectively processing those expensive components later and sending the less intensive bits earlier. With BigPipe, an approach for client-side dynamic content substitution, we can render our pages progressively, where the skeleton of the page loads first, then expensive components such as "songs I listened to most in the last week" or "currently playing" are sent to the browser later and fill out placeholders. This component-driven approach gives us the best of both worlds: non-blocking user interfaces with a brisk time to first interaction and rapid piecemeal loading of complete Drupal pages that leverage the theme layer.

Currently, Drupal 8 is the only CMS with BigPipe deeply integrated across the board for both core and contributed modules—they merely have to provide some cacheability metadata and need no awareness of the technical minutiae. Drupal 8's Dynamic Page Cache module ensures that the page skeleton is already cached and can thus be sent immediately. For example, as menus are identical for many users, we can reuse the menu for those users that can access the same menu links, so Dynamic Page Cache is able to cache those as part of the page skeleton. On the other hand, a personalized block with a user's most played songs is less effective to cache and will therefore be rendered after the page skeleton is sent. This cacheability is built into core, and every contributed module is required to provide the necessary metadata for it.

For a fully decoupled site to load more rapidly than a highly personalized Drupal 8 site using BigPipe, you would need to reconstruct a great deal of Drupal's smart caching logic, store the cache on the client, and continuously synchronize the client cache with server data. In addition, parsing and executing JavaScript to generate HTML takes longer than simply downloading HTML, especially on mobile devices. As a result, you will need extremely well-tuned JavaScript to overcome this challenge.

Client-side frameworks encounter some critical and inescapable drawbacks of conducting all rendering on the client side. On slow connections such as on mobile devices, client-side rendering slows down performance, depletes batteries faster, and forces the user to wait. Because most developers test locally and not in real-world network conditions on actual devices, it's easy to forget that real risks of sluggishness and unreliability, especially due to spotty connectivity, continue to confront any fully decoupled site — Drupal or otherwise.

Progressive decoupling is the future

As we've seen, fully decoupling eliminates crucial CMS functionality and BigPipe rendering. But what if we could decouple while still having both? What if we could keep things like layout management, security, and accessibility when decoupling while still enjoying all the benefits an interaction-rich single-page application would give us? More importantly, what if we could take advantage of BigPipe to leverage shortened times to interaction and lowered obstacles for heavily personalized content? The answer lies in decoupled components, or progressive decoupling. Instead of decoupling the entire page, why not decouple only portions of it, like individual blocks?

This component-driven decoupled architecture comes with big benefits over a fully decoupled architecture. Namely, traditional content management and workflow, display and layout management are still available to site builders. To return to our music website example, we can drag a block containing the songs a user listened to most in the past week into an arbitrarily located region on the page; a front-end developer can then infuse interactivity such that an account holder can play a song from that list or add it to a favorites list on the fly. Meanwhile, if a content creator wants to move the "most listened to in the past week" block from the right sidebar of the page to the left side of the page, she can do that with a few mouse clicks, rather than having to get a front-end developer involved.

We call this approach progressive decoupling, because you decide how much of the page and which components to decouple from Drupal's front end or dedicated Drupal renderings. For this reason, progressively decoupled Drupal brings decoupling to the assembled web, something I'm very passionate about, because empowering site builders and administrators to build great websites without (much) programming is important. Drupal is uniquely capable for this, precisely because it allows for varying degrees of decoupledness.

Drupal 8 is ahead of the competition and is your go-to platform for building decoupled websites. It comes with a built-in REST API for all content, a system to query data from Drupal (e.g. REST exports in the Views module), and a BigPipe implementation. It will be even better once the full range of contributed modules dealing with REST is available to developers.

With Drupal 8, an exciting spectrum opens up beyond just the two extremes of fully decoupling and traditional Drupal. As we've seen, fully decoupling opens the door to a Drupal implementation providing content to a broad range of one or more clients ("many-headed" Drupal), such as mobile applications, interactive kiosks, and television sets. But progressive decoupling goes a step further, since you can fully decouple and progressively decouple a single Drupal site with all the tools to assemble content still intact.

What is next for decoupling in Drupal?

Although Drupal has made huge strides in the last few years and is ahead of the competition, there is still work to do. Traditional Drupal, fully decoupled Drupal, and progressively decoupled Drupal can all coexist. With improved decoupling tools, Drupal can be an even better hub for many heads: a collection of applications and sites linked and supported by a single backend.

One of the most difficult issues facing front-end developers is network performance with REST. For example, a REST query fetching multiple content entities requires a round trip to the server for each individual entity, each of which may also depend on relational data such as referenced entities that require their own individual requests to the server. Then, to gather the required data, each REST query needs a corresponding back-end bootstrap, which can be quite hefty. Such a large stack of round trips and bootstraps can be prohibitively expensive.

Currently, the only way to solve this problem is to create custom API endpoints (e.g. in Views) that comprehensively provide all the data needed by the client in a single response to minimize round trips. However, managing endpoints for each respective client can quickly spiral out of control given the range of different information each client might demand and number of deployed versions in the wild. On the other hand, if you are relying on a single endpoint, updating it requires modifying all the JavaScript that relies on that endpoint and ingests its response. These endpoint management issues can force the front-end developer to depend on work the back-end developer must complete, which creates bottlenecks in decoupled teams.

Beyond performance and endpoint management, developer experience also suffers under a RESTfully decoupled Drupal architecture. For each individual request, there is a single corresponding schema for the response that is immutable. In order to retrieve differently formatted or filtered data according to your needs, you must create a second variant of a given endpoint or provision a new endpoint altogether. Front-end developers concerned about performance limitations desire full control from the client side over what schema comes back and want to avoid working with the server side.

Decoupling thus reveals the need to investigate better ways of exposing data to the client side. As our pages and applications become ever more component-driven, the complexity of the queries we must perform and our demands on their performance increase. What if we could extract only the data we need by writing queries that are efficient and performant by default? Sebastian Siemssen proposes using Facebook's GraphQL (see demo video and project on drupal.org) due to the client's explicit definition of what schema to return and the use of consolidated queries which break apart into smaller calls and recombine for the response, thereby minimizing round trips.

I like GraphQL's approach for both fully and progressively decoupled front-ends. It means that decoupled sites will enjoy better overall performance and give front-end developers a better experience: very few round trips to the server, no need for custom endpoints, no need for versioning, and no dependence on back-end developers. (I even wonder if GraphQL could be the basis for a future version of Views.)

In addition, work on rendering improvements such as BigPipe should continue in order to explore the possibilities given a more progressive rendering system. It's currently in progress to accelerate Drupal's perceived page loads where users consume expensive or personalized content. As for tools within the administration layer and in the contributed space, further progress in Drupal 8 is also necessary for layout management tools such as Panels and block placement that would make decoupling at a granular level much easier.

A great deal is being done, but we could always use more help; please get in touch if you're interested in contributing code or funding our work. After all, the potential impact of progressive rendering on the future of decoupled Drupal is huge.

Conclusion

Decoupled content management is a rapidly evolving trend that has the potential to upend existing architectural paradigms. Nonetheless, we need to be cautious when approaching decoupled projects due to the loss of functionality and performance.

With Drupal 8, we can use progressive rendering to build a Drupal-driven skeleton of the page to fill out increasingly expensive portions of the page. We can then selectively delineate which parts of the page decouple once the page load is complete. This concept of progressive decoupling offers the best of both worlds. Layout management, security, and content previews are unaffected, and within page components, front-end application logic can work its magic.

Drupal 8 is now a state-of-the-art platform for building projects that lie at the nexus between traditional content-rich websites and modern interaction-rich web applications. As always, while remarkable strides have been made with Drupal 8, more work remains.

Special thanks to Preston So and Wim Leers at Acquia for contributions to this blog post and to Moshe Weitzman, Kevin O'Leary, Christian Yates, David Hwang, Michael Pezzi and Erik Baldwin for their feedback during its writing.