The Mandalorian

Jon Favreau turned to video game technology in the filming of The Mandalorian. The Disney+ series uses what’s constantly been referred to as “groundbreaking new technology” for the film industry, Something ILM calls “StageCraft.” Using Fortnite creator Epic Games’ Unreal Engine, the filmmakers render a scene in realtime and then project it onto massive LED screens that encompass the acting space. These scenes are then tracked with pixel-perfect accuracy to the camera, keeping perspective and parallax just right no matter what movements are made.

There are a ton of advantages to this method of filming locations and backplates for scenes. Instead of needing to scout locations or build complex sets, filmmakers can just create the perfect location themselves and have them ready for filming within 24 hours. If adjustments need to be made, such as the positioning of certain elements, coloration, or lighting, they can make those changes on the fly. Time of day also isn’t an issue. The production team described the ability to shoot a 10-hour day at dawn because they can simply set the scene when they want it. No more waiting for perfect weather or lighting.

For the actors, it’s a much more immersive way of filming. The Mandalorian’s creators have talked about almost forgetting that they were on a set when in the middle of a scene. And instead of needing green screens to make post-production edits to the environment, everything in the scene is already captured at the time of filming. This also means that lighting, reflections, and color are more accurate, requiring less post-production and processing to fix those issues, and giving the creators more freedom of choice to make creative effects decisions while filming rather than in the editing process.

Using Unreal Engine and LED screens is a perfect solution for faster TV show production, allowing shows like The Mandalorian to have big-budget sets and locations and get everything filmed and edited faster. The production team continued to expand on the idea throughout The Mandalorian’s first season, even bringing physical set pieces right into the LED screen stage. This added elements of dynamic lighting and reflections off of the physical objects, more physical interaction to the scenes, and allowed them to further blur the lines between what was digitally created and what was actually there on the set.

Sony has announced a pair of new Crystal LED modular direct view displays that could make virtual movie production – of the kind used to create the digital sets in The Mandalorian – a far more widespread method of filmmaking. The displays, developed in collaboration with Sony Pictures Entertainment, comprise several modular panels that use Micro LEDs to produce enormous digital images, allowing actors to exist in virtual movie worlds without the need for traditional green screens. The technology means directors can control set surroundings without moving between locations or constructing expensive building projects, which typically eat into large portions of a movie’s production budget. Sony announced that its B-series displays are geared towards professional applications – including virtual set production – given their anti-reflective coating and high degree of brightness. Sony says the displays can operate at an average of 1,800 nits, which is in the upper range of brightness even for Samsung’s ultra-bright QLED displays. For comparison, even Apple’s Pro Display XDR (referring to its “extended dynamic range”) reaches a peak of 1,600 nits.

Sony’s new displays will allow greater visual fidelity to actors and filmmakers tasked with creating authentic, convincing worlds. It’s the reason why The Mandalorian sets seem so real – the actors are able to deliver their lines while the sands of Tatooine whirl around them, rather than working alongside giant green panels. If actors are convinced of their own existence in these virtual worlds, then so will viewers. Displays like these allow for more natural light emission, doing away with the false lighting required from traditional filmmaking methods.

Reflections, too, are more realistic. Typically, post-production teams are forced to spend hours editing out pesky reflections caused by green screens, inserting false effects to create the illusion of natural light. With Sony’s virtual displays, reflections seen in movies will actually be those caused by the light of the actors’ surroundings, creating a more authentic, immersive visual experience.