Literature Review: Blending the Real World with Virtual Environment

This is a follow-up work after I listened to the talk of RealityCheck @CHI2019. RealityCheck explored how to composite the virtual environment and the real world without pre-captured geometry of the physical world. Combining both the virtual environment and the real world raises a sense of mixed Reality. That is very attractive to me and also researched by us before.

Several approaches have been investigated before. First, using the real world as a blueprint to build up a virtual scene. Reality Skins, as the name suggested, described an approach that the virtual environment is designed based on the floor map and the real world. Because of the reconstruction, Reality Skins are able to provide accurate haptic feedback from the object in the real world. Likewise, our lab has a lab model and roughly model the manipulables like cubes and tables, so users can feel the real world while in VR.

Second, investigating the transition between the physical world and the virtual world for the sake of interacting with the physical world. A dose of reality explored typing only, typing with the virtual environment, and social VR experience with other non-VR users’ existence.

Third, designing various levels of virtuality for the same physical artifact. One Reality designed several levels of visualization and interaction for the same object in the real world. Both projection and headset were used to enable different levels. The levels go from the physical world, augmented surfaces, mid-air content, object decoupling to body decoupling.

Fourth, designing various styles for the same item. That reminded me of what we do to our situational awareness project. NPR tried three visual modes: conventional, stylized and virtualized.

Also, there are some other relative areas, such as the interaction between VR users and non-VR users. ShareVR uses a projector and a large display to show the view from VR users to non-VR users. In addition, it registered the position of the VR user into the large display and the non-VR user into the headset. That maintains the co-presence for all the users.

Very excited about how the virtual environment could be blended with the physical world!