Useful Tips on Working with LED Walls in Mixed Reality

MZed Education

by Mascha DeikovaApril 21^st, 2023

If you are a fan of the Disney+ series, The Mandalorian (which has just aired season 3), then maybe you know how it was made. From the early beginning, the creators of the show shot a vast amount of scenes using virtual production. That’s a great example of where filmmaking technology development goes, and an exciting topic to discuss. In this article, we’ll gather some useful tips on how to achieve authentic-looking footage with LED walls in mixed reality.

One may think that a high-end LED volume stage is only for big Hollywood productions with a major budget. Well, the team from RDM production, which took part in the ARRI Tech Talk series, disagrees. Greg Robbins, Carol Mazzoni, and Chris Fanning used virtual production for commercials and found it to be a genius solution in many cases. Furthermore, there are other types of advanced technology now that make these kinds of tools available for a broader community of creators.

In the video conversation with ARRI Academy, filmmakers share some basic pieces of advice on how to approach a shoot in mixed reality, what mistakes to avoid, and where the power of virtual production really lies. We’ll cover a bunch of their tips below, but if you want to dive more into the technical details – please head over to the ARRI Tech Talk module on MZed.com.

Virtual production – what is it?

First, let’s get the terms right. When someone speaks about a Volume Wall, they essentially mean a bank of monitors that create a huge 270-degree semicircular video wall of screens. Those screens display an interactive background and provide an actual environment for the shoot in real-time. A LED stage works synchronized with the camera’s movement, merging virtual content with the real world seamlessly.

We can think of it as an immersive experience on set compared to shoots on a green screen. In virtual production (or mixed reality – we will use both terms here), the created backgrounds appear as a natural part of the real world and actors can interact with them. Also, there are usually some practical set pieces, combined with digital extensions on the screens.

Preparing LED walls in mixed reality – pre-production process

After this new methodology got noticed, the director Greg Robbins was curious to try it out in their automotive work. Because he and his team already had experience with green screens and had gathered a decent understanding of Unreal Engine (the software they used to create the backgrounds for LED walls), they didn’t have to start from scratch. Still, as the head of production Carol Mazzoni says, the first essential step for anyone who wants to approach virtual production is to do extensive research. In the beginning, it’s not easy at all and it entails a lot of pre-production work that isn’t necessary for a live-action shoot.

Most obviously, you have to prepare those backgrounds in advance. Depending on how you plan your shots, they can be still images, captured videos, and also full 3D environments (either custom-made or from the Unreal Engine store, for example). With the automotive commercial, creators also went out to shoot background plates for the running shots of the cars being driven. The director of photography Chris Fanning recommends filming scenes in the exact same locations as intended, during the exact same time of day, and also – to catch the exact same weather conditions that should be in the final video. This will permit easy, seamless integration of the plate in mixed reality. Look at the example below. Do you see that the car is standing in front of the LED wall? I doubt it.

Backgrounds are one thing, but Carol admits there is a lot more that you have to figure out in pre-production, so be sure to plan an extra budget and time for it. Another example is lighting, which will also mainly come from your LED wall. Here it is important to realize that everything has its own color science (the screens, camera sensor, your additional lights, the processor running the LED wall, etc.) According to Chris, matching these sources to each other is a big part of what helps to sell the footage as being real.

Benefits of virtual production and using LED Walls

As Carol Mazzoni promises: after you’ve done it once, the next pre-production process will go twice as fast. And during the shooting phase, all the advantages of virtual production will finally come to light.

The first obvious benefit of a mixed reality environment is – the crew and cast are able to see what’s in the background (not only greenish material from the green screen), so they can react to it. At the same time, compared to a “normal” live-action shoot on location, you don’t have to move your team and equipment around from one set to another. Production-wise, that means:

“Yes, you have to include in the budget a bunch of other specialists, but not to be worried about the weather, the time of day, the moving of the crew – it’s a dream come true.”

Carol Mazzoni, head of production – a quote from ARRI Tech Talk on MZed.com

Apart from switching locations swiftly, in virtual production, you get neat control over the chosen environments. For example, the DP could, in one gesture, move the sun to the side that he liked most-or create artificial light on the LED wall that would cast stunning and absolutely naturally-looking reflections.

When we speak specifically about automotive commercials, it also means shooting stress-free running shots. The actor and the car are in the studio, so you don’t have to worry about the driver’s safety, distractions that might come into your shot (like police cars, blocking the road for the shoot), or the sudden, unintended shake of the crane. You can concentrate solely on the performance of the talent.

Tricks to seamlessly blend real and virtual footage

However, all those benefits of virtual production come in handy only if you understand how to marry the real foreground to the LED background. The first tip from the cinematographer is to make moving shots and create a real sense of parallax. Typically, you would put the tracker on the camera and let the Unreal Engine create the parallax based on the focal length you’re using.

The LED walls are huge, and only a part of what’s playing there comes into the frame. You can see a square on the wall, which moves and reflects exactly what your lens is capturing. So, it’s possible to select your field of view, your depth of field, and other settings. Then, start blending the real set and the virtual background together.

A good tip from the director Greg Robbins is to also bring movement to the foreground, when possible. It could be a bit of haze, some objects or people coming by, or an actor’s interaction with the real set piece. In the example below, the actress walks on the treadmill and touches the part of the hedge maze (which was created by the art department and then extended digitally in the virtual background), which really sells the trick.

Sure enough, it was important to build the hedge (both practically and digitally), perfectly matching the color and textures, and getting the measurements right. But also during the principal photography, you do a lot of adjustments on the wall itself, says Chris. Brightness, contrast, color temperature, lighting – it takes a lot of time on set to match everything in every shot to make it look real.

What about gear choices?

As Chris Fanning explains, the environments created in Unreal Engine (or similar software) often look very gamey. They’re just too perfect. That’s why, on his first virtual production project, he chose to shoot anamorphic. In his opinion, those lenses add some imperfections back to the footage, which takes this “gaming” touch away and makes it look more real.

In some other projects with mixed reality, he switched to ARRI Alexa Mini LF and Signature Primes. This large format set-up allowed him to gain extra separation between the real foreground and LED background, making the images feel very three-dimensional.

Another tip from Chris to speed up the virtual production immensely is to make sure you get an LED ceiling as well. You can manage without it (as the RDM production team had to in some first projects), but that will mean a lot more supplemental lighting, which has to be creative and match the LED wall. Very hard work, in other words.

Other possibilities to film with LED walls in mixed reality

Nowadays, you can also find other, more accessible tools for virtual production. For example, we once wrote about the work of Quasar Science, the company developing image-based lighting and bringing down the cost for mixed-reality stages. And for sure, there will be more and more LED wall innovations in the near future.

If you want to expand your technical know-how and explore the details about camera gear and lights in the virtual production, we recommend you listen to the full discussion in this ARRI Tech Talk module on MZed.com.

What else do you get with MZed Pro?

As an MZed Pro member, you have access to over 500 hours of filmmaking education (including roughly 17 hours of ARRI Tech Talk sessions). Plus, we’re constantly adding more courses (several in production right now).

For just $30/month (billed annually at $349), here’s what you’ll get:

• 54+ courses, over 850+ high-quality lessons, spanning over 500 hours of learning.
• Highly produced courses from educators who have decades of experience and awards, including a Pulitzer Prize and an Academy Award.
• Unlimited access to stream all content during the 12 months.
• Offline download and viewing with the MZed iOS app.
• Discounts to ARRI Academy online courses, exclusively on MZed.
• Most of our courses provide an industry-recognized certificate upon completion.
• Purchasing the courses outright would cost over $9,000.
• Course topics include cinematography, directing, lighting, cameras and lenses, producing, indie filmmaking, writing, editing, color grading, audio, time-lapse, pitch decks, and more.
• 7-day money-back guarantee if you decide it’s not for you.

Full disclosure: MZed is owned by CineD

Let’s turn the tables now! Have you ever worked with LED walls in mixed reality? What do you think about virtual production in general? Let’s discuss this in the comment section!

Feature image source: RDM / ARRI Tech Talk / MZed.