Sound and Vision

XR experience cannot exist without sound.  Sound is often undervalued in many types of media but in XR environments its role is particularly crucial for designing a truly immersive experience.

 

Some time ago, when I read this paper, I was fascinated by the extent to which vision and hearing complement each other.

 

We hear in all directions but we can only see in front of us.

 

Our ability to detect the exact position of an object visually is much better than our ability to detect the exact position of a sound… However, in time, we are much more sensitive to small changes with our hearing than with our vision — we can hear the difference between two clicks 10s of microseconds apart, while the fastest flickers we can see are around 200 milliseconds apart.

Nevertheless, the most important difference between these senses is that we can close our eyes but we cannot close our ears.  For this reason, I would argue that sound is much more than 50% of XR. Sounds surround us literally every second of our life (even when we sleep). This explains why it is so difficult to create a compelling VR experience without audio. When we see an object that doesn’t match the sound that it makes, our brain automatically knows the object is fake.

 

Of course playing just any sound is not enough — our auditory system is not easily convinced. Sound in VR needs to be rendered according to natural acoustics, and its parameters need to be adjusted to reflect the characteristics of virtual visual layer. This is why 3D audio, plays such an important role in creating convincing sound fields in VR.

 

In the Future Reality Lab, we are working on emerging technologies, and try to answer questions which relate to the future of sound. How do we create a realistic experience which will immerse people in virtual or augmented worlds, and at the same time allow them to interact with each other sonically? How do we think about sound design in a 3D space where listener can freely move around? Finally, what are the new sonic worlds waiting to be explored, enabled by XR technologies?

More on these, and other sound questions in future posts.