When Ready Player One was showing earlier this year, our lab went to watch it together in theater. (Thanks to Ken for buying the tickets!) Ready Player One is a science fiction film tells a story about an adventure in a virtual reality world in the year of 2045. Adventure in VR? It is exactly what we are doing in our lab every day. For a group of VR “insiders” like us, I think Ready Player One is a must watch since it gives us a visualization of the future VR world, makes us think about the trend of the VR technology and probable ethical issues beyond technology. Although Ready Player one is a science fiction adventure, the immersive technology it shows is not unattainable. Full body motion tracking and haptic suit, facial motion capture, slim VR headset, tactical UAV and so on, none of them would be considered as fairytale even in ten years ago. Even though many of them are still in preliminary or prototype stage, a world with all these fantastic technologies is not far beyond reach. When I was watching the movie, I was exited because I believe all the movie showed will definitely be realistic, it let us visually see what is going to happen (only except the ending part that the villain pulled out a small handgun and the crowd fell back on the street of mighty Ohio)
A few weeks ago, Ken gave me a motion capture suit to test. The suit is called Rokoko SmartSuit, which is a wireless and markerless full body motion capture suit equipped with 19 IMUs tracking the major body components. The suit tracks 9-DoF from each IMU, updates at 100 fps, and sends data to a computer through Wi-Fi connection. Unlike traditional IR-camera-based motion capture system such as Vicon and Optitrack, it does not require any cumbersome setup or configuration on a set of cameras, does not need to have a lot of IR reflective markers on certain positions; and more importantly, its effective tracking space is basically unlimited because it does not have to be inside of an overlapped region of cameras. Due to the nature of sensor-based tracking, it is not as good as high-end IR-camera-based MoCap system in terms of accuracy; but in most cases, if we are not doing highly accurate motion analysis, we don’t really need a super high resolution tracking on each body component when we just need to present ourselves with avatars in a reasonable level of accuracy.
(Thanks to Pasan Dharmasena for the video)
However, a full body motion capture system is not truly qualified as “full body” if it is not able to track fingers because our fingers are so important not only in Human-Computer Interaction, but also in our natural interaction. Unfortunately, Rokoko does not track fingers. In order to achieve the real full body tracking, we brought Leap motion into our system. With a Leap Motion mounted on the VR headset, we are able to track our fingers and use the data to drive the left and right hand models of our avatar in VR when we are doing the interaction with the hands in front of our face.
The VR headset we use is Lenovo Mirage Solo, it is a standalone VR display system with a stereo camera in the front. With its inside-out positional tracking, we are able to get our absolute position. The position obtained from Mirage Solo is how we relocate our avatar in VR world when we are physically going around.
(Thanks to Eric Chang for the video)
Given what we have in our lab, we can have this whole system fully untethered. Because the suit and Lenovo Mirage Solo both work wirelessly and Leap Motion just needs a computer, we can connect Leap Motion to a Mini PC or even a slim laptop in a backpack. Besides, the whole system takes less than $4000, it is much more affordable when just comparing to a IR-camera-based MoCap system which usually costs hundreds of thousands. I believe when this unlimited, untethered, wireless full body motion capture solution becomes a consumer product and goes into every family, plus haptic units and facial MoCap added to the suit, everyone can become Wade as in Ready Player One, and I believe this is the ultimate goal in every VR researcher’s mind.
In summary, let me quote a great scientist’s word as the end.
“We call it Ready Player Two”
–Ken Perlin
Very, very cool research going on out there. I cannot wait to see where this goes in the coming months. I am interested in some of the applications you have been using with your motion capture suit and Mirage Solo headset. Are there any immersive experiences / games that take advantage of the hardware combo you are describing?
Thank you for your comment and following our blog!
We currently have a demo as the first Gif shows. On the screen in the Gif, you can see that we built a 3D model of our lab whose scale in VR matches its physical scale. When the player goes around the physical space, the avatar goes around in the VR lab accordingly. Because this is done recently, we haven’t applied it to any other games or experiences yet. But the step is to put it in our Siggraph VR project if it fits. And some of my colleagues are making games here, we will put it in games when they need full body tracking.
Pingback: Fixing Mocap – Future Reality Lab
Pingback: Make Limited Space Unlimited – Future Reality Lab