AI Compositing with NVIDIA BROADCAST inside Unreal Engine

What is Nvidia Broadcast?

Nvidia Broadcast is one of few software’s which provides a AI driven compositing solution to users. Nvidia Broadcast allows users to apply effects over a in-coming camera feed, in this demonstration we used the “replace background” feature allowing us to get a composite without the need of any greenscreen; the way Nvidia archives this ability is by harnessing the power of both machine learning image analysis and depth perception.

Once we have our composited green background in Nvidia Broadcast we move forwards to streaming the feed into Unreal Engine. From the real-time engine we have the ability to place the talent into any 3D environment and now aren’t limited to the previous 2D dimensions.

See below our tutorial and demonstration video:

Relevant links and software’s:

Whats Next?

Moving forward, this proof of concept has proved itself to work and now needs officially testing back in our OSF lab on a GODBOX™ with a 4K camera feed input.

 

One Response

  1. Ignacio Pol says:

    There’s even a better way to this: in the Nvidia Broadcast app select Background Removal (instead of Replacement). With this option you have a perfect alpha on your background ready to send to OBS (you will see a black BG but trust me, there’s alpha there). In OBS you you have to add the Spout Output as a filter effect in the Video Capture Device layer. Finally in UE4 you can use the video texture coming from Spout as a regular texture with alpha (you can make a material and apply it to a plane for example), just create a Masked or Translucent Material and connect the alpha of the texture to the Opacity slot in the material. The same applies to a Composure material, using RGBA to blend between video and CG with the Over node. This way is better because Unreal doesn’t have to chromakey anything and you save that process. Hope this helps, if you have any doubt, don’t hesitate to contact me and I’ll be glad to guide you step by step.

Leave a Reply

Your email address will not be published. Required fields are marked *