Using Motion Capture in Video Production

Zimmy, a buoyant teenager from the group Turf Feinz does his street dance in a jump suit lined with LED markers. A PhaseSpace motion capture system acquires Zimmy’s street dance movements which are mapped to a virtual Zimmy.

In the Direct Images Interactive web video production New San Leandro, we showed how a VACTOR (virtual actor) was created and placed in a virtual world.  In the latest promo video, San Leandro: Prototype City of the Future, I thought it would be an interesting challenge to place the virtual actor into the real world. Well, almost a real world… a live-action video world that is.

To achieve a camera move where the virtual character is moving about within a real world, live-action video, we needed to attach LED markers not only on Zimmy, but on the camera as well. By tracking the points of the camera movement, we would utilize a virtual motion control system.  If a camera shooting a real world “background plate” is locked down so that the environment is stationary, this would be a much easier composite to achieve than a shot with a moving camera.

The shot was to be just a few seconds, and involved a few performers who would be watching the avatar move about.  It was to feel as if a drone flew into PhaseSpace to see a group of young people discovering what innovation and creativity was all about.  We set up a camera boom to simulate the drone movement and used a Canon 5D Mark III to shoot the “background plate” of the extras.

Once we had our video master, we still needed to composite the avatar in.  When the recorded motion capture information came back, it was not a video, but rather a series of data points which referenced the talent position, the camera position and the movement.  Working with engineer Kan Anant, we proceeded to render to video a basic avatar against an alpha channel (invisible background). At first I wasn’t sure if I wanted to layer the avatar on top of Zimmy as an overlay or have Zimmy perform his moves off camera and just composite the moving avatar onto the moving vieo.  I ended up using the shot with Zimmy in it, but rather than cover him over, the avatar was offset to make it look like his dancing partner. It seemed more engaging and fun to have the avatar dancing side by side with Zimmy in tandem which is what we ultimately decided upon.

Producer Derick Lee and I were on a mission to introduce community members in the marketing video production and to present a discovery of cutting edge San Leandro businesses. The act of discovery was not only for the viewers of the video, but was for the participants in the making of the video itself.  It was great to work with Zimmy and see his talented kinetic moves and to learn the nuances of the PhaseSpace motion capture system from Kan.