Can Drupal outdo native applications?

I've made no secret of my interest in the open web, so it won't come as a surprise that I'd love to see more web applications and fewer native applications. Nonetheless, many argue that "the future of the internet isn't the web" and that it's only a matter of time before walled gardens like Facebook and Google — and the native applications which serve as their gatekeepers — overwhelm the web as we know it today: a public, inclusive, and decentralized common good.

I'm not convinced. Native applications seem to be winning because they offer a better user experience. So the question is: can open web applications, like those powered by Drupal, ever match up to the user experience exemplified by native applications? In this blog post, I want to describe inversion of control, a technique now common in web applications and that could benefit Drupal's own user experience.

Native applications versus web applications

Using a native application — for the first time — is usually a high-friction, low-performance experience because you need to download, install, and open the application (Android's streamed apps notwithstanding). Once installed, native applications offer unique access to smartphone capabilities such as hardware APIs (e.g. microphone, GPS, fingerprint sensors, camera), events such as push notifications, and gestures such as swipes and pinch-and-zoom. Unfortunately, most of these don't have corresponding APIs for web applications.

A web application, on the other hand, is a low-friction experience upon opening it for the first time. While native applications can require a large amount of time to download initially, web applications usually don't have to be installed and launched. Nevertheless, web applications do incur the constraint of low performance when there is significant code weight or dozens of assets that have to be downloaded from the server. As such, one of the unique challenges facing web applications today is how to emulate a native user experience without the drawbacks that come with a closed, opaque, and proprietary ecosystem.

Inversion of control

In the spirit of open source, the Drupal Association invited experts from the wider front-end community to speak at DrupalCon New Orleans, including from Ember and Angular. Ed Faulkner, a member of the Ember core team and contributor to the API-first initiative, delivered a fascinating presentation about how Drupal and Ember working in tandem can enrich the user experience.

One of Ember's primary objectives is to demonstrate how web applications can be indistinguishable from native applications. And one of the key ideas of JavaScript frameworks like Ember is inversion of control, in which the client side essentially "takes over" from the server side by driving requirements and initiating actions. In the traditional page delivery model, the server is in charge, and the end user has to wait for the next page to be delivered and rendered through a page refresh. With inversion of control, the client is in charge, which enables fluid transitions from one place in the web application to another, just like native applications.

Before the advent of JavaScript and AJAX, distinct states in web applications could be defined only on the server side as individual pages and requested and transmitted via a round trip to the server, i.e. a full page refresh. Today, the client can retrieve application states asynchronously rather than depending on the server for a completely new page load. This improves perceived performance. I discuss the history of this trend in more detail in this blog post.

Through inversion of control, JavaScript frameworks like Ember provide much more than seamless interactions and perceived performance enhancements; they also offer client-side storage and offline functionality when the client has no access to the server. As a result, inversion of control opens a door to other features requiring the empowerment of the client beyond just client-driven interactions. In fact, because the JavaScript code is run on a client such as a smartphone rather than on the server, it would be well-positioned to access other hardware APIs, like near-field communication, as web APIs become available.

Inversion of control in end user experiences

Inversion of control animation
When a user clicks a teaser image on the homepage of an Ember-enhanced Drupal.com, the page seamlessly transitions into the full content page for that teaser, with the teaser image as a reference point, even though the URL changes.

In response to our recent evaluation of JavaScript frameworks and their compatibility with Drupal, Ed applied the inversion of control principle to Drupal.com using Ember. Ed's goal was to enhance Drupal.com's end user experience with Ember to make it more application-like, while also preserving Drupal's editorial and rendering capabilities as much as possible.

Ed's changes are not in production on Drupal.com, but in his demo, clicking a teaser image causes it to "explode" to become the hero image of the destination page. Pairing Ember with Drupal in this way allows a user to visually and mentally transition from a piece of teaser content to its corresponding page via an animated transition between pages — all without a page refresh. The animation is very impressive and the animated GIF above doesn't do it full justice. While this transition across pages is similar to behavior found in native mobile applications, it's not currently possible out of the box in Drupal without extensive client-side control.

Rather than the progressively decoupled approach, which embeds JavaScript-driven components into a Drupal-rendered page, Ed's implementation inverts control by allowing Ember to render what is emitted by Drupal. Ember maintains control over how URLs are loaded in the browser by controlling URLs under its responsibility; take a look at Ed's DrupalCon presentation to better understand how Drupal and Ember interact in this model.

These impressive interactions are possible using the Ember plugin Liquid Fire. Fewer than 20 lines of code were needed to build the animations in Ed's demo, much like how SDKs for native mobile applications provide easy-to-implement animations out of the box. Of course, Ember isn't the only tool capable of this kind of functionality. The RefreshLess module for Drupal by Wim Leers (Acquia) also uses client-side control to enable navigating across pages with minimal server requests. Unfortunately, RefreshLess can't tap into Liquid Fire or other Ember plugins.

Inversion of control in editorial experiences

Inversion of control editor
In CardStack Editor, an editorial interface with transitions and animations is superimposed onto the content page in a manner similar to outside-in, and the editor benefits from an in-context, in-preview experience that updates in real time.

We can apply this principle of inversion of control not only to the end user experience but also to editorial experiences. The last demos in Ed's presentation depict CardStack Editor, a fully decoupled Ember application that uses inversion of control to overlay an administrative interface to edit Drupal content, much like in-place editing.

CardStack Editor communicates with Drupal's web services in order to retrieve and manipulate content to be edited, and in this example Drupal serves solely as a central content repository. This is why the API-first initiative is so important; it enables developers to use JavaScript frameworks to build application-like experiences on top of and backed by Drupal. And with the help of SDKs like Waterwheel.js (a native JavaScript library for interacting with Drupal's REST API), Drupal can become a preferred choice for JavaScript developers.

Inversion of control as the rule or exception?

Those of you following the outside-in work might have noticed some striking similarities between outside-in and the work Ed has been doing: both use inversion of control. The primary purpose of our outside-in interfaces is to provide for an in-context editing experience in which state changes take effect live before your eyes; hence the need for inversion of control.

Thinking about the future, we have to answer the following question: does Drupal want inversion of control to be the rule or the exception? We don't have to answer that question today or tomorrow, but at some point we should.

If the answer to that question is "the rule", we should consider embracing a JavaScript framework like Ember. The constellation of tools we have in jQuery, Backbone, and the Drupal AJAX framework makes using inversion of control much harder to implement than it could be. With a JavaScript framework like Ember as a standard, implementation could accelerate by becoming considerably easier. That said, there are many other factors to consider, including the costs of developing and hosting two codebases in different languages.

In the longer term, client-side frameworks like Ember will allow us to build web applications which compete with and even exceed native applications with regard to perceived performance, built-in interactions, and a better developer experience. But these frameworks will also enrich interactions between web applications and device hardware, potentially allowing them to react to pinch-and-zoom, issue native push notifications, and even interact with lower-level devices.

In the meantime, I maintain my recommendation of (1) progressive decoupling as a means to begin exploring inversion of control and (2) a continued focus on the API-first initiative to enable application-like experiences to be developed on Drupal.

Conclusion

I'm hopeful Drupal can exemplify how the open web will ultimately succeed over native applications and walled gardens. Through the API-first initiative, Drupal will provide the underpinnings for web and native applications. But is it enough?

Inversion of control is an important principle that we can apply to Drupal to improve how we power our user interactions and build robust experiences for end users and editors that rival native applications. Doing so will enable us to enhance our user experience long into the future in ways that we may not even be able to think of now. I encourage the community to experiment with these ideas around inversion of control and consider how we can apply them to Drupal.

Special thanks to Preston So for contributions to this blog post and to Angie Byron, Wim Leers, Kevin O'Leary, Matt Grill, and Ted Bowman for their feedback during its writing.

Comments

Taco (not verified):

This presentation at DrupalCon New Orleans was great. First time I heard a crowd explode into applause during a DC talk :)

We have an intern working the next 5 months on progressive web app for our Open Social distro. Main goals will be push notifications, add to home screen and offline capabilities. As always we will share the practical results with the Drupal community!

Dries:

What technology stack are you planning to use? Keep us posted on what you learned and thanks for you contributions.

Andrej Galuf (not verified):

I think the second paragraph nails the whole thing perfectly: Native applications seem to be winning because they offer a better user experience on the devices they're ment to be used on. Ethan Marcotte coined the term "Responsive Web Design" in 2011, yet in 2016, RWD is still a novelty in many places, something that's sold as a special extra on top of "a website". How, then, can open web fight off feature (and input) rich apps?

If you ask me, Rich experience with Inversion of control (be it Ember or otherwise) should be a basic standard. Nothing less should be accepted and nothing less should be offered. Not the day after tomorrow, but today. Yesterday, even.

anon (not verified):

Completely different methodology. Most developers don't seem to prefer how Angular 2 works, those doing Angular 1 development aren't very rosey on it either.

anon (not verified):

I am on the fence about officially adopting any external framework into core seeing how we already pack in jQuery and Backbone. Backbone was the most popular framework at the time, and the dev cycle / decision to use it / put in core took so long that by the time Drupal 8 was released, virtually no one talked about doing Backbone development in a long time and almost a year later (since 8.x release) no one writes about it at all, even in the Drupal world.

Can that same guarantee be made for the 'next' thing to add to core? Or would effort have to occur to convert out and remove Backbone specific code? The ember demos and the talk at Drupalcon New Orleans was impressive, but I fear that it gets added 12-24 months from now and Ember has fallen out of favor with devs the same way Backbone had. Every front end dev out there seems very "React" crazy, even though it doesn't do all the things Ember does.

These are just my reservations.

But the overall point is, Drupal needs something. It needs the first class API, and the slick, user-friendly AND dev friendly interface on top. Drupal 9 will be too late. Nearly every known CMS in the marketplace, free or proprietary, are racing toward this goal.

Dries:

This is a valid concern. For simplicity's sake, let's say we have two options: (1) shipping core with Backbone or (2) shipping core with Ember, Angular or React? Both options could be suboptimal. What option would be the least bad from a developer's point of view?

Bruno Mateus (not verified):

I understand your point-of-view, but maybe there are other choices.

Why a full commitment with one single framework? In the end, many of the new frameworks solve 90% of the same problems - but with different strategies, semantics and methods. Maybe there could be a way of exposing an universal, standardised API that could interface Drupal with the "external world" (read "not drupal"), bridging to any framework we choose with the Drupal backend.

It could turn Drupal even more flexible, allowing more choice and future-proof to the developers. No-one knows the js "flavour of tomorrow" - they change on a monthly basis - and to become locked in a specific "flavour" may not be the best move. Besides, there's always someone that won't like the chosen "flavour", and won't be keen on learning that specifically.

I've personally become a fan of decoupling Drupal, as I can use different frameworks for different types of work, using REST and web services. It's not for every project, there's problems to address and a lot of work to be done, but also wins to be reaped.

Nonetheless, yours is a very much needed discussion.

Andy Hails (not verified):

This is really interesting stuff. I am a big fan of the decoupled approach and the power of Drupal to potentially utilise the latest frontend technologies. It could be ember, react or whatever - Drupal could be the engine behind whatever vehicle we choose.

This has nothing to do really with the article so not sure of this is the place to raise it but as I was reading about the ember demo, I was intrigued by an idea: what would the Drupal Editor UX be like if we removed the "edit" page for nodes (or at least Removed it as default). In place editing would be the only method for users creating and updating content. Switching between view modes to see how your content would look in certain scenarios (this would also encourage component based theming). Even have a mode to see what your mode would look like when exposed as JSON through a rest service - editable even.

My question is, with such good frontend tools now providing slick and stable UX is there still a use for the node/%/edit page?

Dries:

I might be misunderstanding your proposal, but I believe there might always be a need or preference to keep the traditional edit page. In a world where Drupal is a content repository that outputs content to different devices in different formats, in-place editing is not always an option. For example, Vodafone uses Drupal to power in-store kiosks. These kiosks are effectively TVs and don't support in-place editing.

Andy Hails (not verified):

Sure of course, that's one of the most powerful parts of Drupal - moving towards a device agnostic approach. The Vodafone example is a good example for this. This is all "blue sky" thinking so take what I say with a "pinch of salt" and probably for another forum!

I'm thinking about this in the context of the outside in Drupal. Yes certainly for the Vodafone example the edit page seems like the best approach (you wouldn't imagine an editor editing JSON) but for other, let's say "more traditional" web layouts the "edit" page screen could be seen as archaic. Right now we present users with two paths - edit and quick edit. Why? The later, for me, seems to hinder the outside in approach. Why give the user so many options if they don't necessarily need it?

dalin (not verified):

This stuff looks nice and fancy, but the problem I see is that there's really two tiers of Drupal websites being created right now:
- Fully decoupled sites (those with big budgets, maybe 5% of all new sites)
- Traditional fully coupled sites (the 90% majority)
- And then a couple who are straddling the line.

I don't have anything to back up these numbers, but I'm confident in the rough ratios.

In order to pursue a fully decoupled approach you need a big team(s): back-end devs, site builders, front-end framework developers, plain-ol' HTML/CSS front-end devs, devops, and lots of PM to manage multiple teams. All this can only be done with big dollars. This is somewhat related to the "Drupal is for ambitious web projects" discussion, though it's perfectly possible to make an ambitious fully-coupled web project with a team of 1.

This separation concerns me. I'd only support a front-end framework in core if there was a significant focus on bringing in that 90%. Drupal 8 has only had marginal, or even negative progress in making things better for lower-level developers. So we'd be fighting against history to make this happen.

I also second others' concerns about bundling a front-end framework that becomes obsolete in 2-3yrs.

Dries:

Dalin, I share your concern about keeping Drupal accessible but I don't think your comparison is fair. You only need a large team when there is a lot of development to do. Today, most fully decoupled applications are custom built from scratch, and therefore require larger teams. If Drupal were to ship with a decoupled front-end and/or back-end, you might not need to do a lot of development -- let alone custom built something from scratch. While things might get a bit more complex, it might still be completely feasible to build a site with a team of 1.

Marco Galimberti (not verified):

Some (chaotic) thoughts on that:
I think that the difference between native and web applications it’s just a fine line, it depends on the point of view… everything on the web can be executed locally if something gives the proper layout to show data, and REST services in core solve (or will solve) a big part of the problem.
The ExtJS framework has some nice example (http://examples.sencha.com/extjs/6.2.0/examples/) that shows complex layouts embedding data with JSON calls (via REST services). Latest versions of this framework works on a MVVM model (https://www.sencha.com/blog/ext-js-5-mvc-mvvm-and-more/) and it’s possible to define a data structure to get needed data. I think that this framework it’s very heavy (lot of files and code have to be loaded the first time and performances are not very good) and the license model changes (almost) every major versions, but it shows that the difference between native and web application it’s really a fine line (see the desktop example)…
So, if it’s possible to render data in such a way, what are the other components involved in the process to make a web application a native one?
I think that browsers are the key… Mozilla’s XULRunner is a package of components to bootstrap XUL/XPCOM applications based on javascript code (among other), get data on the server (even if the javascript framework doesn’t support data models) and access OS libraries (via XPCOM) to use OS features not included in browsers. Some example such as xulapp-starterkit (https://github.com/racklin/xulapp-starterkit) shows how to do it.
Some other examples are NW.js (http://nwjs.io/) and Electron (http://electron.atom.io/).
Other interesting projects which embed this technology in a bigger context are Chromium and Firefox OS which include Rich Client Platforms (RCP) in a dedicated OS, but this exceeds this post.
To have a Drupal dedicated client on the phone with these technologies we just have to configure a website, download JS and CSS of the website and check periodically for file changes (e.g. with a checksum) if we want to shows the server’s layout, or develop some scripts to generate the data model on the client side (that should be provided by the server).
Done! :-) ...we now have a dedicated Drupal client compiled for every Operating System. I hope this is helpful.
Marco

zhilevan (not verified):

So what about angular or React?
there are many method and technologies that we can integrate drupal with them, as example decoupled method introduced and used by lullabot team "https://www.lullabot.com/news/announcing-the-new-lullabotcom" ,
the situation is complicated , what is the best way ?
it seems Reactjs kill anuglar http://en.arguman.org/react-is-killing-angular , but which one is really better comes in drupal core?

it seems react is the winner of client sides javascript frameworks.as a drupal developer I scare of future drupal,

anon (not verified):

You cannot declare react "the winner" with no backup or comparisons.

Michael Keara (not verified):

Thank you Dries, this is an interesting post about an important topic. Inversion of Control is a key part of an Outside-In approach to creating and managing web sites. It's important to keep in mind that an Outside-In UX does not only apply to managing content (as most examples seem to focus on) but also the *site-builder* user experience.

Site-builder tasks can benefit significantly from an Inversion of Control UX strategy. To really see this we need to keep in mind that any solid UX design is based on a clear definition of task sets that exist to achieve specific goals. (In my UX design background I refer to these task sets as 'Roles' but the Drupal community uses that term to refer to sets of permissions. A given logged-in user with a fixed set of permissions can switch Roles repeatedly as they take on different goals - e.g. modify content, change a layout, build a View, etc. The real value behind a task-oriented definition of Roles is the direct relationship between a Role and the UI tools required to get that task done. I think it's safe to say that simplicity of UX is tied to having the right UI tools at the right time - no more, no less.)

The risk of approaching UX design that is based on Inversion of Control is that, without a solid, task-based UX strategy, the resulting product could be limited to a set of disparate, unrelated tricks rather than a fully capable user interface system.

In the days of the Spark initiative (when the 'outside-in' concept was first emerging in the Drupal community) I put together an envisioning video to explore how this approach could be applied beyond content editing tasks and make an impact on the site-bulder UX itself. Looking at the examples given here I see significant similarities so I want to invite anyone interested in Outside-In to take a look at that video again. It has three parts:
1: a look at the overall problem and a quick look at the Spark approach
2: a 'right-brain' exploration of how Outside-In could apply to site-building
3: a schematic look at the structure of an Outside-In site-building system based on the types of context (dynamic, static, relationship) that underly Panels, Page Management and Layout.

I believe the concepts in that video still apply, perhaps more than ever, and this could be useful to breaking down the challenges and help establish a strategic approach to an Outside-In / Inversion of Control UI system. It's a 15 minute video so grab a coffee, relax and enjoy!

You can view it here: https://www.youtube.com/watch?v=T2U_AyM5ePM