How augmented reality can be used to superimpose product information

Last spring, Acquia Labs built a chatbot prototype that helps customers choose recipes and plan shopping lists with dietary restrictions and preferences in mind. The ability to interact with a chatbot assistant rather than having to research and plan everything on your own can make grocery shopping much easier. We wanted to take this a step further and explore how augmented reality could also improve the shopping experience.

The demo video above features how a shopper named Alex can interact with an augmented reality application to remove friction from her shopping experience at Freshland Market (a fictional grocery store). The Freshland Market mobile application not only guides Alex through her shopping list but also helps her to make more informed shopping decisions through augmented reality overlays. It superimposes useful information such as price, user ratings and recommended recipes, over shopping items detected by a smartphone camera. The application can personalize Alex's shopping experience by highlighting products that fit her dietary restrictions or preferences.

What is exciting about this demo is that the Acquia Labs team built the Freshland Market application with Drupal 8 and augmented reality technology that is commercially available today.

An augmented reality architecture using Drupal and Vuforia

The first step in developing the application was to use an augmented reality library, Vuforia, which identifies pre-configured targets. In our demo, these targets are images of product labels, such as the tomato sauce and cereal labels shown in the video. Each target is given a unique ID. This ID is used to query the Freshland Market Drupal site for content related to that target.

The Freshland Market site stores all of the product information in Drupal, including price, dietary concerns, and reviews. Thanks to Drupal's web services support and the JSON API module, Drupal 8 can serve content to the Freshland Market application. This means that if the Drupal content for Rosemary & Olive Oil chips is edited to mark the item on sale, this will automatically be reflected in the content superimposed through the mobile application.

In addition to information on price and nutrition, the Freshland Market site also stores the location of each product. This makes it possible to guide a shopper to the product's location in the store, evolving the shopping list into a shopping route. This makes finding grocery items easy.

Augmented reality is building momentum because it moves beyond the limits of a traditional user interface, or in our case, the traditional website. It superimposes a digital layer onto a user's actual world. This technology is still emerging, and is not as established as virtual assistants and wearables, but it continues to gain traction. In 2016, the augmented reality market was valued at $2.39 billion and it is expected to reach $61.39 billion by 2023.

What is exciting is that these new technology trends require content management solutions. In the featured demo, there is a large volume of product data and content that needs to be managed in order to serve the augmented reality capabilities of the Freshland Market mobile application. The Drupal community's emphasis on making Drupal API-first in addition to supporting distributions like Reservoir means that Drupal 8 is prepared to support emerging channels.

If you are ready to start reimagining how your organization interacts with its users, or how to take advantage of new technology trends, Acquia Labs is here to help.

Special thanks to Chris Hamper and Preston So for building the Freshland Market augmented reality application, and thank you to Ash Heath and Drew Robertson for producing the demo video.


JvE (not verified):

So, how hard is it to have Drupal manage the images in the Vuforia Cloud? Or does the store have to manage the article data separate from the images? And if you have to manage the images at Vuforia anyway, why not use its content management to store the metadata (up to 1mb metadata is included there)?

Chris Hamper (not verified):

I don't yet have personal experience integrating a Drupal site directly with Vuforia's Cloud Recognition service in this way, as the Acquia Labs proof-of-concept I helped develop did not demand that kind of integration. Vuforia does have a well-documented Web Services API, so it should be relatively straightforward to have Drupal sync images to a Vuforia target database. This would require some custom code, as there aren't any Contrib modules I've come across that provide this functionality.

It's definitely an interesting idea to store all of the product information within each target's metadata field. This would have the benefits of the product info coming along with the target when it is identified, and taking load off the Drupal site. However, Vuforia itself doesn't provide a Content Management System; it only stores targets. Vuforia doesn't supply a robust interface for creating and modifying structured data associated with a target. As a result, managing this product information solely through the services Vuforia provides would not be practical.

Drupal does provide this content management functionality. Also, using Drupal to manage your product content comes along with the ability to easily integrate products with other functionality on your Drupal site; for instance recipes, store inventory, and product locations inside a store. It seems like it might be a viable option to push an encoded copy of a product's data in the Vuforia cloud target database each time a product is added or changed on the Drupal site. You'd have to decide if the benefits of this outweigh the downside of complexity for your own use-case. There also may be Vuforia API limits you could bump into with this approach, if you have to update products frequently.

Add new comment

The content of this field is kept private and will not be shown publicly.

Plain text

  • No HTML tags allowed.
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.