Shooting LED in camera VFX
The challenges you’ll face on-set when shooting LED in-camera VFX.
We all know UE4 with LED technology is rapidly securing its place on-set, it’s green, its fast, it’s good, but boy it’s expensive, so let’s get ready for more LED on-set and talk about some of the challenges of shooting LED in camera for final pixel.
Since the Mandalorian behind the scene set photos appeared online people have been asking “What equipment and LED did the team use on the Mandalorian?” Here’s a quick round up of what equipment was used and then look at some of the challenges.
The Equipment On-Set
Looking behind the LED walls used on Mandalorian
On the Mandalorian Cinematographer Barry “Baz” Idoine was able to see and shoot LED screens as real-time virtual backdrops in Unreal Engine (that could switch to being a massive green screen at the click of a mouse) in this post we’ve summarised and expanded on some of the reported production challenges Baz and the team faced while filming LED through the camera. Originally reported in ICG Magazine, we expand some thoughts on the challenges, some thinks to think about.
“ILM then partnered with Epic [Games] to make their Unreal Engine for gaming into a robust production tool,” Bluff adds, “allowing for real-time display on LED screen walls.”
The best LED pixel pitch for the camera, on this size stage, with the camera this far away from the wall, was deemed to be 2.8 Talking to other filmmakers testing LED 2.8 seems to be the popular starting pitch for LED virtual film sets. It is also the pixel pitch of the LED OSF are testing in our labs. But why is this the case, why is this becoming the go-to panel pitch? We spoke to industry insiders about this and they said that some previous attempts by filmmakers to use much lower pitch (1.8, 0.9) failed saying the 2.8 pitch works for large sets, with the lower pitch panels more suited to smaller sets and for use in close-up reflections.
Lower pitch may not be best for virtual backdrops.
Colour shift with moving cameras was also reported to be problem in the ICG article. At the lower pitch ranges (2mm or lower) the LED panels physical pixel is a very small dome or cylinder that sits on top of the panels surface. As you move your position the light of each pixel can shift with your viewing (camera) angles, creating a shift in colour. Think about how your TV or monitor has a viewing angle, it’s essentially the same with LED.
Solving the colour balance problem is a case of testing and making sure all your system components are set to run on the same colour space.
LED moire issues, seeing moire effects through the camera.
Test test test would be our advice on all of this. You want to make sure your LED pixel pitch is in tune with where you expect your camera to be. If you can get hold of a panel to test with, put it on set and put a subject in front of it, now focus on your subject. If you depth of field is to big, using say a zoom with a small aperture, you may get moire issues coming from the LED. Use a fast fall -off lens, wide open and a narrow depth of field.
Using LED for reflections
Highly reflective surfaces such as plastics or glass, can still be tricky as a high pixel pitch panel can be seen as a pixelated reflection if too close to the subject and especially if the lens pulls sharp focus on a reflection and the panel is not far enough away from the subject. The solution to this is have some very high pixel pitch panel available on set to pull in for these situations.
The higher the LED pixel pitch the lower the light on-set
Looking at one individual pixel, the smaller the LED pixel is (to achieve the lower pixel per inch pitch resolutions (of 1.4, 0.9 etc) the less light it emits, or at least on camera it seems to travel less distance, throws less light on the subjects.
But what you do get is what in VFX or games design terms we’d call a HDRI global lighting effect. With an LED stage you get an amount of lighting that gives a global look to your subjects. You can then come in with on-set lights to add fill and highlights or flag off any unwanted light.
Blending the different processes for an in-camera solution required many vendors. Profile Studios provided camera tracking, lens (FIZ) data, and alignment of the physical and virtual worlds. “The camera tracking and lens data were streamed simultaneously to three Unreal Engine [UE4] workstations and a StageCraft workstation, each operated by the ILM stage [Brain Bar] team,” reports Profile president and creative director Matt Madden. see full article.
OSF Director of Virtual Production Asa Bailey shoots on virtual sets and builds VP systems, he adds “ More people need to test LED, importantly test results need to be shared. But my big question is how do we democratise LED knowledge and hardware so more people can create with it?”
Good question, we are working on it OSF.
The bigger the pixel the more light it puts out throwing more light on your subject.
Focussing on reflections may see pixelation if pixel pitch and distance don’t compliment.
The pixel pitch resolution of the LED screens used on this Star Wars project was 2.8.