Last week was very busy for me as our lab was preparing a Concert on the Holodeck. The event featured Ozark Henry – a very talented Belgium avant-garde pop artist. The performance was part of the Holodeck project, which aims to build a Virtual Reality platform for different kinds of collaboration.
This time we connected Ozark Henry on the stage with musicians in Norway and a choir in the Dolan recording studio. Besides that, the remote dancers were dancing to the music in the FRL. Their movement was captured by the Optitrack system and applied to the moving avatars. The video was then streamed to the main screen in the concert hall. The audio between different spaces was broadcast live using JackTrip and MPEG-H technology – a new type of audio codec which allows the encoding and decoding of any immersive audio format.
Our research on musical collaboration in VR is at the beginning stages and many questions still need to be answered. Designing a VR system for music collaboration raises several challenges which have to be taken into account, including:
- Latency – during live performance, musicians can hear each other almost instantaneously, the only delay they experience is propagation delay (the time needed for sound to reach from the instrument to the ear). Having too much latency makes it difficult or even impossible to perform some kinds of music. The delay tolerance depends on the genre of music, musicians’ skills, instrumentation and tempo, and varies between 20-50 ms. If the delay exceeds these values, different performance approaches need to be applied.
- Joining different acoustic spaces – when connecting musicians from two different acoustic spaces (i.e. concert hall and acoustically treated studio) it is important to make them feel as they are sharing the same virtual acoustic space. Should we impose the acoustics of one space onto another? Should we create an artificial space which will be shared by the artists? How much accuracy of the reverberation (in the sense of realism) is needed? How does this influence the performance? These are the questions which need to be answered.
- Localization – each of the musicians has their own location in space which should be reproduced on the receiver’s side. 3D audio technology allows to reproduce sound source in any direction around the listener but implementing it raises challenges, particularly since the more the number of channels are required the more the conversion time and latency can increase.
- Visuals – eye contact is a very important factor as it helps musicians to communicate during the performance. It is not clear yet, what is the best way to present musicians in VR and how much details of their movement is needed to make the performers comfortable.
As you see we have so much to explore!
If you would like to know more, contact Andrea from the Music Lab as he is writing a dissertation on this topic.
Source article:
https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7769205