Augmented Audio

Something I’m really excited about the VR and AR mediums is the development of spatialized audio. I love sound designing, especially for animated films, because it’s about world building. In a film, it’s not the visuals that tell you what you’re supposed to feel at any moment, but the soundscape — the music, the foley sounds (that’re specifically chosen for a scene), the intonation of what characters sound like, and so much more. In a crash crash scene, often times thunder rolls or lion roars are mixed in to trigger the primal reactions to those sounds in our unconscious minds, to add intensity into a car crash. There’s a saying (and it differs slightly between individuals) but: The visuals account for 30% of the experience, the rest depends on the audio. The soundscape of a scene often tells you what’s happening in the world of the film outside of what the camera is showing you. Even in reality, the hi-resolution portion of our vision is only about 1-2 degrees (about the size of your thumbnail at arms-length). On the other hand, our auditory perception can’t be shut off — the brain is constantly processing the sounds around you — and we’re able to extract a lot of information from it — to the degree that you can tell which direction, its distance, and the size of the space around the audio source.

So, now considering VR – a medium where you’re not able to dictate exactly what you want to show to the audience. The sound of an experience becomes infinitely more important, because it’s now used to direct people’s attention without being too intrusive.

This leads me to binaural audio. Taking advantage of how our brain processes audio, we can recreate the feeling directionality + distance to a much higher degree than stereo audio. (Barbershop example – LISTEN WITH HEADPHONES) And this is the standard for creating VR audio.

But, as awesome as headphone binaural audio is, there is a level of the audio experience that is impossible to recreate with headphones. The ability to feel the sounds — the audio experience you get when you go see a concert or movie. And something that’s really cool right now in the world of audio development is spatial audio speaker systems. There’s a few techniques out that I know of are wavefield synthesis and ambisonic audio. (If you want details, you’d have to talk to Martha). There’s a really cool company out in Berlin called 4DSound creating a setup of pillars of omnidirectional speakers to create spaces where sound can be shaped. (Spatial Sound Hack Lab – the 4DSound guys hosted a session for musicians and technologists to come in and create experiences with their system). Imagine listening to a song and instead of hearing a drop, you have to walk into it. The idea with mixing in the 4DSound system is that you can give a shape to a sound – ie. a sphere of 2 feet or a wall of this sound at the edge, and an individual can’t hear these sounds unless they walk into it.

Now, what happens when binaural audio and spatial audio speaker systems are combined. I hadn’t really considered this until our CAVE experience at Siggraph, where the pre-show was an audio experience using a pair of Bose off-ear headphones with an array of 4 speakers in the room to create the soundscape of the Ice Age. And our mainshow had a subwoofer that played back the mammoths roar + footsteps.

When thinking about creating in-person mass VR experiences, since there’s a physical space requirement, I think it’s really important to rig these spaces with audio systems that can further enhance the audio world of narratives told in VR. With binaural audio, you’re able to give everyone a unique sound experience to each individual while the speaker system can aid in further cementing the ‘reality’ of the virtual world in people — as well as, a tool to emphasize emotional moments in a narrative.

An awesome resources on learning more about sound design:

(They also have amazing breakdowns of soundscapes of films and other areas of audio recording/mixing for films.


(Update: GO READ Marta’s article on presence and audio here)

Leave a Comment

Your email address will not be published. Required fields are marked *