Green Screen in Pass-Through Video

When HTC Vive released their ultimate mixed reality (MR) Vive Pro, we were all surprised that a front-facing dual camera is integrated into the headset. Users can see a stereo passthrough video by each eye obtaining a separate image from the stereo camera.

Ken told me about one of his idea that is having a green screen hang up in the physical world, and the green screen pixels in the video passthrough are replaced by the pixels of a rendered computer graphics scene. It means that the green screen is like a window for us to see the virtual scene if the virtual scene is placed behind the image planes of the passthrough video.

Here is a video showing how it works. The green pixel replacement is not working perfectly. Because the green screen is so thin that it is sort of semi-transparent, plus the lighting condition was not controlled. There are a lot of new experiences to implement after this green screen experiment verified that we can edit the passthrough video itself in addition to just rendering 3D objects with it when creating an MR experience.