Realtime Cinematics Motion Capture and Animation in Unreal Engine
With our partners at Mo-Sys Engineering, Xsens and Unreal Engine we wanted to produce something at the furthest end of the virtual production spectrum; a fully CGI realtime rendered virtual production featuring realtime motion capture, virtual cinematography, and Hollywood grade camera motion controls, with no post production allowed (except an edit in Premier).
Leading the shoot was Director of Virtual Production Asa Bailey, who’s team created some quick sets and character assets, mainly bought from the Unreal Marketplace. But then we wanted to bring in something a little different to martian landscape.
We had 500 images of an archaeological dig site of an ancient skeleton, and we liked the idea of this very real and ancient character making an unexpected appearance in a virtual production set on another planet that only exists in virtual space. We pumped the skeleton dig images into our RT3 Workstation running our photogrammetry software and after some minor clean up in Maya we exported the skeleton as a photorealistic UE4 asset.
What you see in this video is purposely low-fi in terms of post production (you could do so much more), in fact there is no post production, everything is acted out on-set, recorded, rendered and controlled in realtime, except for laying the shots down on a time line after the shoot, adding some music and graphics all in Adobe Premier.
We used our own designed and built RT3 virtual production workstations equipped with 2 x NVIDA RTX Titan GPU, i9 CPU, 64gig of RAM all cased in what is essentially fridge, worked like a dream when outputting the shots.
Realtime Motion Capture was taken care of by the Xsens suit and the Mo-Sys Startracker Belt which when combined create a rock solid realtime motion capture solution with no slipping.
The point of the test was to optimise our on-set virtual production pipeline for realtime animation and realtime motion capture to see how good we can get it – out of the box so to speak. Our aim is to start using this set up and technique for full CG game Cinematics and creating fully immersive VR films and narrative.
The possibility to post produce the shots is still there, it’s just for this test we wanted to see what we could do in a very short amount of time. We filmed in the greenwich Mo-Sys Lab for one day and we got allot of shots from our virtual camera rigs, recording directly into Unreal Engines Sequencer.
With all the takes saved in the UE4 project, all we had to do was open the project, boot up the saved takes in sequencer and record out the virtual camera moves in UE4’s native 4K AVI files, we output at 75fps, as our final video timeline was set up at 25fps.
Production equipment list:
OSF RT3 Workstations
Unreal Engine 4.22
MO-SYS Hollywood Hand Wheels
Xsens Mo-Cap Suit
Mo-Sys Director Viewfinder
Mo-Sys VFX for UE Plugin