Storytelling Across Multiple Mediums

How often is a new creative medium made in one’s lifetime?

We’ve had three new mediums that’ve been enabled through many recent breakthroughs in technology over the last decade — virtual reality, augmented reality, and creating living/reactive physical spaces via IoT (Internet of Things) devices — for individuals to express themselves and create art in.

Since I was getting photo/video coverage of CAVE over the course of Siggraph 2018, I spent a lot of time watching people going through the line, pre-, main-, and post-show. Something became clear to me over the course of those days — that this could be the beginning of a new form of experiences people could easily go to in the very near future. An event series that utilizes all the emerging technologies and combines with the arts to tells stories across multiple mediums, by transporting our audiences into different realities. An opportunity that used to be fairly restricted to large theme parks and given to individuals like the Imagineers at Disney to make, will be more accessible and easily incorporate into spaces outside of amusement parks.

 

Entertain this next idea with me:

Like we go to movies to watch stories be told on a big screen or immersive theater, like Sleep No More, to live through a surreal spectacle. We could go to events and be transported through different realities.

Pre-Show

The moment you step in line or the waiting room, you step into the experience — you’re a part of a journey. With the miniaturization of IoT sensors and devices, a space could be designed to react to a multitude of different things (ie. movement of people, audio-reactivity, etc). As audiences interact with each other in the room, they can effect the space they’re standing in. Lights/projectors and a spatial audio setup in the room could be designed to tell the pre-show part of a larger narrative told over the course of the event — a prologue and introduction into the world where the story takes place. The interactivity built into creating a living space would guide audiences toward a state of flow or state of suspension of disbelief— where they’re more readily to accept the reality of the narrative they’re a part of.

Pre-show Sidenotes: Why have a pre-show? To get groups of strangers to begin bonding with one another before going on a virtual reality journey together. This will give individuals a chance to get excited about the experience they’re about to go through, as well as someone to talk to about the experience afterwards. Some design considerations to think about in addition to narrative ideas, is how the interactivity built to trigger the visuals and spatial audio can effect human behaviors — like how to get strangers to interact with one another? Or how to move them naturally throughout the space without the need of docents ushering people? 

Photo: Guy dancing in front of a full wall projection while wearing Vive trackers to alter the visuals. (Panorama Festival: Lab 2018, Superbright)

Photo: Physical Portal (waiting room) leading into the Flatland dome film (Panorama Festival: Lab 2018, Magenta Field)

 

Main-Show

Next, as the audiences move into the main-show, they step into a dome where the projections and spatial audio rig with an ambisonic mix mimics the experience of being at a location from the narrative. In a large enough dome, the projections can create the illusion of standing inside any environment — outdoors or indoors. As the audience straps on the VR headset, they themselves get transported into the world they just saw projected in the dome.

Inside virtual reality, we’re able to tell a story to a group of people by taking them on a journey through many spaces. The language for this medium is still in its infancy, just as it took 100+ years to develop the storytelling language we utilize in films (wide/close-up shots, montages, etc.), we’re at the first year of this immersive medium — not to mention the ways this virtual reality medium can be used in tandem with augmented reality and IoT enabled spaces to tell a story across mediums. Exciting times right? All the possibilities! 🙂

An idea that I hadn’t really considered before working on CAVE was the concept of creating augmented layers to the audio / sound design experience. After seeing the Bose prototype eyewear headphones, which doesn’t cover your ears directly but shoots a highly directed stream of audio right into your ear, it clicked in my head that designing shared virtual reality experiences needs to take advantage of both the binaural mixing + head-tracking from VR headsets with an array of speakers + subwoofers placed throughout the room. The speaker rig needs its own layer of sound design / music mixed utilizing techniques of wave field synthesis to create an even more visceral audio experience. Headphones are great to make sure the audience hears quiet sounds / procedural sounds that are custom to an individual’s behaviors (ie. sounds that can be triggered by gaze as one diverts their attention to their surroundings), but the speakers + subwoofer layer allows for emphasis of moments where people are able to ‘feel’ the physicality of sounds, especially low frequency and loud sounds. Something I really wish we have opportunity to do in the future, is to have the VR audience sit inside a circular array of stacked speakers at different heights and a grid of speakers over everyone’s heads to experiment with designing layered audio experiences for a virtual reality experience.

Mainshow Sidenotes: As human beings, we experience the world through our senses, and we’re prone to a lot of different illusions. So, how can we hijack our senses to strengthen the illusion of existing in the world of the story? In building the main-show, something to consider is how to use multiple senses to tell a story. (ie. using aroma diffusers in the rooms, creating a realistic soundscape with the audio setup, and taking advantage of perspective illusions in domes or full room cube projections to alter the size of spaces.)

Photo: Girl grabbing a ‘virtual’ particle (CAVE @ Siggraph 2018, Future Reality Lab)

Then as audiences take off their headsets, the dome projections can adjust to reflect the end of the virtual reality story, so that as everyone takes off their headset, they’re still stepping back into the familiar world that they were just inhabiting.

Photo: Portal in Flatland Dome Film (Panorama Festival: Lab 2018, META)

 

Post-Show

As audiences exit and re-enter reality, with where augmented reality is headed, the characters, props, and places can be brought out into the larger world. Seeing what Holojam has done with procedural character animation, little apps can be created for people to easily make their own custom animated creations using assets from the experience and put it out into the world via AR — so people can share with others a glimpse at what they experienced or to tell a post-show story.

Post-show Sidenotes: Some ways to design this experience could be to think of it like the end credit scenes to Marvel movies, a way to tease audiences with something more. Or it could be used as a memorabilia from the event – characters, props, and location assets that can be used to make something new.