The Quest for Cheap and Easy Facial Capture

As we started spinning up production for CAVE, our massive SIGGRAPH show, we were immediately faced with a challenge: how do we go about animating the main character’s face? One idea was to find out if we could get access to a motion capture rig that could do facial capture, but that would limit all capture to a few short sessions, and would potentially be expensive. Another idea was to use a depth camera like the Intel Realsense, but we quickly learned that while there are fantastic cameras on the market, we could not find one that had a pipeline for going from a face in front of a camera to Maya or Unity animation.

.
Then, CAVE’s technical artist, Thomas Meduri, directed me to an awesome YouTube video. In it, the developer was showing off how he could puppeteer the body and, could you believe it, the face of one of his models, using a self contained mocap suit and a helmet with a phone hung out like a carrot on a stick.

.

The developer had decided to use the iPhone X for facial capture, piggy-backing on Apple’s ARKit. The moment I saw the video, I was determined to see if we could do the same. And we could!

.

.

After spending a day working with Unity’s ARKit Plugin, we were able to puppeteer a model straight off of the iPhone. Then, we faced an even bigger hurdle: how to save out the data? Unity doesn’t have anything built in for animation saving, and a Unity .anim file != a Maya .anim file. In the end, I went with an old but true route – a custom file writer for both Unity and Maya. That way, the files could be written and distributed in real time – no need for any post-processing. A further step was to bring it all into an app, and have it distribute the files through a file sharing system like Google Drive.

.

.

Finally, there was one more feature we absolutely needed – audio capture off of the phone’s mic, so that we could use it in post for synchronizing face to other, higher quality recordings of the voice. With that, we had everything we needed, and could get usable, good quality data in multiple file formats.

.

There’s a few more features I want to implement beforehand, but in true FRL fashion, we intend to release this app publicly in the near future. Stay tuned!