Real-Time In-Camera VFX Could Be the Green Screen Future

NewsSoftware

by Jeff LochAugust 16^th, 2019

During the Unreal Engine User Group at SIGGRAPH 2019, the team at Unreal Engine showed an impressive real-time in-camera VFX technology that could change how films are made. While this term is often overused, this technology could be a game-changer and kill the green/blue screen in the near future. Let’s take a closer look.

Unreal Engine

Unreal Engine is a game engine developed by Epic Games which is out since 1998. Indeed, you can use it to build video games like Fortnite. But, you can also use Unreal Engine’s real-time technology for architecture purposes, VR and AR, and even cinematography.

The Unreal Engine is an eco-system of tools and applications. The entire source code is available for free, so everyone can modify it and expend the various engine features. If you are not a “code guy,” don’t worry. It’s easy and intuitive to try and create whatever you want in the Unreal Engine without having to write a single line of code.

The kind of photorealistic graphics or videos you can pull out of the Unreal Engine are stunning.

Image credit: Unreal Engine.

Real-Time In-Camera VFX

The team at Unreal Engine has revealed an exciting collaborative project with Lux Machina, Magnopus, Profile Studios, Quixel, ARRI, and Matt Workman. I like to see people like Matt Workman involved in this kind of project. To make it short, Matt is a cinematographer and founder of Cinematography Database. Also, he is the creator of Cine Designer, which is one of the most potent pre-production and lighting tool for DPs available on the market.

Matt Workman during the Unreal Engine User Group at SIGGRAPH 2019. Image credit: Unreal Engine.

During their Unreal Engine User Group event at SIGGRAPH 2019, the whole team experimented around real-time in-camera VFX. Here is the setup:

1. A studio with four giant LED panels (three walls and a ceiling) to act as a background.
2. All four LED panels are powered by nDisplay to render the Unreal Engine Project simultaneously on all four displays.
3. Various engine-controlled ARRI SkyPanels all over the studio. That way, the team can match in real-time the lighting of the “real subject” and the lighting of the background.
4. A couple of sensors on the camera that are linked to the Unreal Engine. That way, the Unreal Engine 3D background moves accordingly to the camera position (and focal length) in the space.

The results in the video at the beginning of this article are impressive. The actor and the props are perfectly blended into the environment/background. All of that happens in real-time in-camera without the need for green screens.

Image credit: Unreal Engine.

The End of Green Screen Shooting?

The beauty of this system is that everything in the background is CG. If you don’t like something in the scene, or something is not in the right place for a specific camera angle, move it out of the way. In Unreal Engine’s example, you can see how easily they pick a rock and displace it in real-time.

However, you could also do it in post-production if you shoot your film on green/blue screen, right? It’s true, but it means you would have to modify your 3D scene and render it again.

Image credit: Unreal Engine.

The main advantage of this technology is that everything is happening in real-time in-camera. It can save you a tremendous amount of time in post-production. However, the major drawback of this technology is that once the shooting is over, you can’t change your mind on what’s happening in the background.

In this “prototype” stage, I think that this Unreal Engine technology could be helpful for commercials or short lengths projects. Of course, it’s not ready for multi-million dollars feature films, yet. But the VFX future looks exciting for sure.

What do you think of this Unreal Engine real-time in-camera VFX technology? Do you think it could replace green/blue screen shooting in the next years? Let us know in the comments down below!