There were already many posts about the Cave, but I have yet to write about the design process which led us to create the prologue to the main show.
The Cave was not only a VR experience. The first thing the audience experienced when entering the project space was an audio soundscape.
Our goal was to create a kind of sound painting which would lead the audience into the world of the Cave. Whenever I start working on sound design for a specific scene, I try to imagine myself in the place and focus on what I would feel, think, see and hear in that environment. In the Cave, we took the audience to the ice age, in the middle of a windstorm with snow blowing in their face. They could hear mammoths and other animals escaping from their homes, and groups of people migrating with their children. I wanted them to hear and feel the struggle that comes with strong wind and cold. The best way to achieve this was to use binaural audio.
The work on sound design started with gathering sound effects of animals, humans, and winds from several databases. Next, we recorded foley in Dennis’ studio. Most of what we recorded were breaths from different perspectives, coughs, and cloth rustling during movement. When all layers were prepared and mixed, we recorded binaural tracks using the 3DIO microphone.
3DIO is a binaural microphone which allows us to capture sounds in a way which is very close to how our ears hear them. Each sound which we hear comes to our two ears with different levels, time and spectral content. Using a model of the human head and placing microphones inside the artificial ears, it is possible to capture sounds with these differences. The sound captured has a unique realism, which is not possible to achieve with stereo recordings.
We used an acoustically treated studio to record the binaural audio. To mimic the movement of animals around the listener, we moved the microphone in front of the speaker which was playing sound. We recorded each sound separately so that they can differ in direction and speed of movement.
As a last step, we mixed all of the files together and added necessary processing, including reverb and delay to achieve a “mountain echo” effect.
We used Bose’s Prototype Eyewear Headphones for audio AR as our main playback system. They provide high fidelity sound and they do not occlude ears. In this way you could hear a mammoth and at the same time discuss its size with someone next to you. Ryan Shore composed a music track which was played through three speakers in the room. The goal was to create an immersive atmosphere where the audience could not tell if the music was coming from their headphones or from the speakers.
I have to say that I was not sure how the experience will be received in SIGGRAPH, as the main focus of the conference is vision. A lot of people were asking: so where are the visuals? (Which makes sense at the world’s biggest computer graphic conference). But there were also people who stayed and listened for a long time.
Tip: Dennis found a really good plugin for creating audio particles: http://soundparticles.com/dopplerair.html.
We used it to create the effect of snow blowing in your face. I suggest looking into this tool for anyone interested in sound design.
I also recommend listening to this recording which shows the capabilities of binaural technology: https://www.youtube.com/watch?v=8IXm6SuUigI