Unreal Engine Helps Asa Bailey Deliver Real-Time Virtual Production
Asa Bailey took the role of Director and DVP (Director of virtual production) on the short film series created for NTT Data. Each short film took just 4 days to produce; Asa was able to create such rapid productions with a pioneered new Unreal Engine based real-time Workflow. In this article we’ll walk you through this real-time ready Workflow and how virtual production took the production to new heights.
The series was created for a company called NTT Data, one of the largest IT services providers in the world with more than 120,000 employees globally. The series was created to show the future direction of NTT Data within each different industry sector, with industry’s spanning from automotive to healthcare. The series focused on individuals being sent to the future to be awakened on the future of their company. The clients wanted a very sci-fi aesthetic to the series and we archived this through fully virtual sets throughout the production.
The pre-production on a real-time project like this was crucial, Because of the huge cuts in post production with virtual production methods we have to compensate with pre-production. There was a continuous back and fourth between Asa and the 3D team. Deciding animations, sets and colour schemes. The short films harnessed virtual and physical objects, meaning there was still a strong need for the arts department. In our workflow we keep anything touched or held by the talent a physical object, creating a augmented production between the real and virtual worlds.
When on-set Unreal Engine really came into its own, supporting the production with its real-time composites and powering our virtual production system which used a group of software’s including Unreal Engine, Brainstorm’s infinity set and chroma hardware’s.
The virtual production system allowed us to previsualise the entire production in real-time, showing the talent composited within the virtual sets. This was then shown all over the studio allowing the crew and actors to see an idea of the finished product. The system also records in 4 different layers, a composite of just the talent on black, the optical layer, the chroma key and a final composite which is talent and virtual set. This allows total freedom for the production it gives the ability to go down the traditional VFX route with just the optical and a pre done chroma key, or what we did was take 90% of shots straight out of the system and into editing with little touch ups in post.
Another element of the real-time on-set production is the actual camera and optical layers. It’s essential that the virtual and real world lightings match, if there’s a flashing lamppost in the background of the virtual set that needs to be reflected on the real talent.
Next Step for Virtual production
New virtual production methods and technologies are constantly surfacing and with the rapid growth of Unreal Engine by Epic Games, virtual production is constantly expanding and providing new opportunity.
The integration of real and digital characters is a massive leap for virtual production. We are now able to have virtual characters and physical actors in the same environment at the same time, whether that environment is virtual or a physical set the digital character can be inserted to either. We can now do this through new real-time facial and motion capture.
As seen in the video below Mo-Sys engineering are bringing this vision into the present through their Unreal Engine plugin and 3D tracking. Here you can see how Mo-Sys bring together tracking, Unreal engine and motion capture to present a almost reversed virtual production with the only virtual asset being the character whilst everything else remains real from the studio to the presenter. The performance was all done in real-time.ARVE Error: src mismatch
src in: ?dnt=1
src gen: https://player.vimeo.com/video/359419724?dnt=1vimeo compared
src gen: https://player.vimeo.com/video/359419724
Real-Time Lighting and Reflections
Lighting within virtual production can be a massive challenge but now we have the ability to adjust virtual and physical lights in real-time to get the perfect lighting on both, the real and virtual sets. This allows total lighting symmetry with the virtual world creating a more believable finished product.
The development of LED walls is rapidly growing with pitches moving from 2.5mm to 0.7mm in less from 5years. With a 0.7mm pitch on the LED screens its possible to shoot them as backgrounds and provide a totally real-time in-camera VFX option. With normal green screen shoots green spill is a issue for all CG artists but when using LED walls the spill from the screens is actually the perfect lighting for the virtual scene.