Work

About

Contact

I designed a volumetric video platform. It runs all volumetric, photogrammetry, and 360 assets, and serves as a white label solution for clients to develop their own XR apps. 

Role: ui/ux designer 
DURATION: ongoing 

At a Crossroads:
Augmented Reality & Volumetric Video

Volumetric video is here – how will consumers enjoy it? As spatial tech, it complements with augmented reality (AR) very well. With more than one billion ARKit/ARCore devices around the world and the ubiquity of AR on social media apps, mobile AR is perhaps the most direct path to  consumer adoption of the volumetric format. I designed a white label AR volumetric player for iOS and Android that runs on Unity.

i. what kind of experiences will live through VOLUMETRICS? 

We looked at simple applications for consumers: People will want to take selfies with celebrities, have a pro athlete show them how to work out in their living room, record it, and share it with friends. From  observations of how users behaved with existing content, I described two broad personas for early adopters of volumetric content: 

The Socializer places holograms in choice locations and situations to take photos and videos, and sends them to his friends. He is young, tech savvy, and familiar with using AR in social media apps. Volumetric AR is not only entertaining, but is a vehicle for self expression and connection. 

The Student enjoys demonstrative holographic experiences. She wants to see a lesson by her favorite yoga instructor, or a fashion model walk across the room in . She is health and image-conscious, and is looking for content that connects with her interests in fashion,  beauty, and fitness

Fitness instruction, dance, fashion advertising, and more - lifelike volumetric videos add a new level of presence and immersion

ii. WhAT WILL THESE EXpERIENCES ALL share?

 These consumer experiences, social and personal, share the same core principles as other AR experiences – to be highly interactive and immersive while delivering a seamless experience . In addition as a young and novel format, volumetric experiences should be something that users desire to share, whether or not that is the primary end-goal of a specific experience. 

III. translating core principles into features

With these principles, we wanted to build a player with  core features and settings in place for future clients and dev teams to build their games on, whether their target user is a “socializer”, “student”, or someone else entirely.

 These core features and their behaviors reinforce what we’ve found to be best practices when introducing new users to volumetric media.

  •   Intuitive placement and playback of holograms in AR
  •   Simple live editing of the AR scene - lighting, colors, effects.
  •   Photo, video recording, and share functions

IV. DESIGNING AND TESTING

I used Adobe XD and Figma to design several iterations of the Player, and fleshed out each interaction using Protopie.  I worked next to a team of Unity developers that implemented the UI and interactions.

We modified features based on bug reports and tester feedback, and  by testing how the player handled in the wild. How fast can I position holograms? How easy was it to make adjustments to take pictures or record video? How is battery life performance?

“Going on assignment” to capture moments with the Player in various environments allowed me to hone in on interactions that were reliable and easy to use. (And that indie AR film I made in IKEA? Yeah bet you’ve never heard of it)

You can play with the ProtoPie interaction prototype here. 

placement and tracking for volumetrics

Problem: The most standard method of AR tracking requires the user scan the room for surfaces, but is inconsistent due to a variety of environmental factors. It fragmented the experience of browsing and playing volumetrics with the repetitive and frustrating task of scanning.  

Solution: Taking inspiration from Google Playground, we automated the surface scanning and placement, although this in turn reduced the reliability of surface tracking. 

Early gesture tutorial popups
New - less is more

let users take it out of the box, from 3d to ar

Problem: Throughout all stages of testing, users continued to get stumped by AR placement when surface detection took more than a couple seconds. Sometimes the hologram would appear off-screen, or more jarringly, life-sized right in front of the user. 

Solution: By loading the hologram in a 3D environment first, users instantly have a controlled space to interact with the hologram. We also found it facilitated that “aha” moment, when a first-time user understands the concept and possibilities of volumetric technology. 

how can we make the player more ergonomic?

Problem: Users were trying to to use the app one handed, and also regularly switched to landscape orientation. However, there was no one-handed method to place holograms, and the UI/interactions did not respond to use in  landscape orientation.

Solution: I added a rotate button and analog scale slider – these replaced gestures requiring two offhand fingers with just the thumb of the holding hand. Users can now navigate and take photos one-handed. I created landscape mode UI that retains the ergonomics of taking photo/video and scrubbing through video.

NOW WHAT?

The mobile player is a straightforward experience on phones and tablets; immersive use cases are limited for the stationary 2D screen. A VR player opens up a new realm of opportunities such as live theater and musical viewing experiences, as well as the RAM to run several volumetric assets at once.  

With the mobile player, I will continue to explore interactions that come with different types of content and use cases – such as changing play speed, inspection pop-ups, and dialogue.  

I designed a mobile volumetric video browser. It runs all volumetric, photogrammetry, and 360 assets, and serves as a white label solution for clients to develop their own XR apps.  

As both videos and 3D objects, how are volumetric videos best presented and navigated in a browser? How will publishers film and organize their holograms? How will users select, preview, and download holograms, and manage the photos and videos they take? 

I designed a white label mobile browser of volumetric and spatial content. It consists of design components that versatile enough to be handed off to other teams to quickly adapt and develop into their own product. I stayed close to Google’s Material Design guidelines throughout the process. 

 

Organizing for versatility

Problem: How do I design a folder structure that is navigable as a general content hub (“Netflix of Volumetrics”), and detailed enough for another publisher to easily adapt for their own app? 

Solution: I structured folders by Series, Episodes, and Scenes –  the holograms themselves.  Episodes are mid-level folders – like seasons on a TV show – and are condensed into tabs for smoother browsing. 

I was inspired by UI of mobile apps for Netflix, Youtube, and Sketchfab, a platform for browsing 3D models. My white-label design incorporated   aspects these content browsers. 

THUMBNAILS DIFFERENTIATE FOLDERS AND SCENES

Problem: What should be the standard design of thumbnails for volumetric content, as both videos and objects?  

Solution: Thumbnail dimensions quickly differentiate between folder and scene. Folder thumbnails are vertical as they display the hologram in a neutral, usually standing pose. Each thumbnail consist of two label fields (title, duration/category), and two image files (foreground, background).  

Eight thumbnail states states indicate the interaction states or status of the volumetric file. 

 

double tap to download

Problem: Volumetric file sizes are very large, and there is not yet a reliable streaming solution on the horizon. Getting users past the Download button is a major challenge – how do you deal with user hesitation and impatience when delivering their assets? 

Solution: Ongoing. There’s no easy UX answer to this, other than fantastic content. We implemented the Holothumb (below). I tried implementing clever gestural interactions that encouraged users to download several files at a time, but they tested to be counter-intuitive – I kept Download as a simple button tap. We also reduced video quality options to prevent analysis paralysis.

A HOLOTHUMB FOR YOUR PATIENCE

Overcoming large file sizes was a major challenge of the content browser.  At the scene page, I replaced the still thumbnail with  a “Holothumb” of the hologram that informed users (“Do I want to invest 500MB into this?”)  with than a loading bar. We found that some measure of instant gratification made for a smoother experience. I designed Holothumb stages – simple themed Unity scenes for every character consisting of a skybox and floor measurements (meters). 

a gallery for sharing

Problem: How will users access their photos and recordings of their holograms? 

Solution: All photos and videos are stored in a local device folder. A gallery is accessible via hamburger menu. The gallery allowed users to add text and stickers and share their media. While we found many users preferred to export their files and edit in Instagram or Snapchat, having in-app editing features  in place would make it easier for future teams to create their own branded experiences.

what next?

  • Design of Desktop (WebAR) and VR content browsers
  • Expansion of social features support – user profiles. 
  • UI for sideloading, importing, and downloading content from outside links