By KATIE KASPERSON
By KATIE KASPERSON
When Industrial Light & Magic (ILM) debuted its StageCraft system on Disney+’s The Mandalorian, the game was afoot: virtual production had officially hit the mainstream. Filming in front of a curved LED screen (dubbed “the volume”), DP Greig Fraser substituted physical locations for virtual environments, rendered instantaneously in Epic Games’ Unreal Engine. These high-resolution backdrops offered the additional benefit of image-based lighting, allowing Fraser to capture visual effects in-camera – something that historically required clever tricks like rear projection or scale models.
In the years that followed, companies raced to build LED stages, develop camera tracking technology, implement Unreal Engine and test the bounds of what this filmmaking method could do. Once sequestered to post, visual effects specialists became a key part of virtual production, their work intertwined with that of cinematographers, set designers and the like.
Though unintentional, DPs have found themselves at the center of virtual production – particularly those working in sci-fi, fantasy and other VFX-heavy genres. Today’s hybrid DP blends traditional cinematography with virtual technology, navigating complex tools and increasingly collaborative workflows while maintaining their original goal: to shape a film’s visuals and tell a story.
While The Mandalorian’s success brought virtual production to the fore, this filmmaking technology was arguably overdue. Before spearheading volume work on Percy Jackson and the Olympians, Pierre Gill was experimenting with virtual location scouting on an unreleased Assassin’s Creed game installment. Working under Ubisoft’s direction, Gill explored the video game from within, plucking out assets as he went along. “The goal was to do what we eventually did with the volume,” he says. “We just weren’t there yet.”
Virtual production had come a long way by the time ILM released StageCraft. Oren Soffer, who served as Co-Cinematographer on The Creator, first worked alongside Fraser during pre-production on The Mandalorian. “I was fortunately exposed early to virtual filmmaking,” he begins. “On The Creator, thanks to Greig, we shot two environments on ILM’s StageCraft, using it for both lighting and in-camera background rendering. ILM then enhanced those backgrounds during post-production, which was mostly necessary on wide shots.” Soffer observed as Fraser liaised with the film’s virtual art department. “Greig prepared them to use ILM’s StageCraft at Pinewood Studios,” where they shot for the final two weeks before wrap. “It taught me so much about virtual production – a skillset I’ve taken with me on a number of projects since,” Soffer says.
Some cinematographers, including Soffer, find virtual production tools relatively easy to understand. “I’ve been a casual gamer my entire life,” he explains. “I found using VR tools to be somewhat akin to playing a video game. From there, picking up the concepts and getting used to the workflow – collaborating with the virtual art department, the on-set VFX department and the physicality of hybrid environments on LED volumes – felt fairly intuitive.”
The same goes for Vanja Černjul, DP on House of the Dragon, its third season slated for release this summer. “As a cinematographer, virtual production does come instinctively. You don’t necessarily have to understand the nuts and bolts of it; I don’t know how all the tools work, but there’s always somebody who does. As long as I know what looks real and how to judge the background from the foreground, it’s the same thing we always do,” he states.
Joining House of the Dragon on for its sophomore season, Černjul can attest to how far virtual production has come – even in two years. “What we did on Season 2 feels like ancient history compared to what’s available now,” he explains. Using an iPad [dubbed “the Cyclops” by the team] “allowed us to monitor how the real set interacted with the virtual set. We used the Cyclops to orient ourselves in a scene; in Episode 207, the whole set was basically a virtual space.”
On Season 3, Černjul spent large chunks of time in what he called the “Tech Vis Room,” a space run by the VFX department. “We worked with this previs system, which had every location scanned,” he describes. This included a “big open exterior” where Černjul could use cranes. “I didn’t want to limit the director by putting a large piece of equipment in the way of a potential shot, and there was nowhere to hide. I could go [into the Tech Vis Room] and – knowing which shots we wanted to get – place the camera and therefore determine where the cranes could go and how I needed to move them so they would never get in the way.” Černjul adds, “They couldn’t get me out of that room once I realized what I could do there. It’s amazing how much planning you can do ahead of time and how prepared you can be when you come to set.”
Catherine Goldschmidt, who’s been with House of the Dragon since the beginning, used an LED volume during pickups and reshoots on Season 1. “We had a specific time of day we wanted to hit, in a really long scene as well, that we couldn’t possibly get during the magic hour window,” she recalls. “We used the volume to good effect. On Season 2,” Goldschmidt continues, “the volume stage got torn down, and the sets that had previously been volume sets were turned into greenscreen sets. That decision was pure economics. [Following] a TV schedule, they couldn’t finish the effects in prep. That meant that the whole premise of shooting in-camera was not met. They were spending time and money in prep to attempt to have in-camera effects, and then they were still spending time and money in post to fix those effects.”
Despite her credits on VFX-heavy projects like House of the Dragon, The Last of Us and 3 Body Problem, Goldschmidt works more with green and bluescreens than LED stages. Instead, she primarily uses virtual tools during pre-production and for image-based lighting. “What we tried to do on 3 Body Problem, which is something that Greig Fraser has been a big proponent of, was use Unreal Engine to previs specifically how the lighting would look on set. We were designing the lighting into the set, and we wanted to make sure we were getting it right,” she says.
“There was some talk in prep that was like, ‘You’re trying to previsualize your lighting in Unreal. Does that mean you won’t need to do any physical pre-lighting on the set?’ Absolutely not,” Goldschmidt clarifies. “Once the set was built and the lighting was installed, we still had to do what we always do – set levels, set color, and all of that had to occur in the real space. It was just a tool for us to see our ideas ahead of time.”
When it came time to light, Goldschmidt considered using LED screens to achieve an interactive effect. “We talked a lot in prep about what would be best to create that light,” she recalls. “The VFX department was pushing for us to get a volume wall. They weren’t trying to finish any effects in-camera, but they wanted – and I wanted – to have that light be as close to the finished thing as possible.” Eventually, Goldschmidt and her gaffers conducted “a comparative test between a volume wall and a Sumosky, which is basically a ladder of long LED strips. We wound up going with the Sumosky because it produced better light and was a lot cheaper.”
On other productions, such as House of the Dragon, image-based lighting is the clear answer. “It was especially important because our characters were all wearing armor, so getting the proper reflections helped create the illusion of reality,” Černjul says. Mostly, though, he used virtual production during prep. “It helped me do my own thing – not integrate visual effects into cinematography, but just to plan. I would basically create virtual storyboards.”
He’s not the only one. Caleb Heymann, Lead Cinematographer on Seasons 4 and 5 of Stranger Things, used Unreal Engine to plan complex, stunt-heavy scenes, such as the ‘oner’ in Episode 504 “One of the biggest sets [on Season 5] was the MAC-Z – the militarized downtown Hawkins,” he begins. “The VFX team created that set as an Unreal Engine asset; that was instrumental in getting everybody on the same page.” Heymann and his camera crew “came together before construction and were able to walk around the set virtually and put on different lenses to see our exact field of view,” he recounts. “We were able to figure out where construction cranes, soft boxes and bluescreens were going to be. It’s amazing what you can do in there.”
During prep, Heymann would consult with the VFX department nearly every day. “On a project like Stranger Things,” he explains, “the line is continually blurred from where our department ends and where VFX begins. It’s something we constantly discussed, and there’s a huge overlap.” Despite the post-production required – particularly on the series’ later seasons – to bring the Upside Down to life, “there’s still a philosophical commitment to doing everything in-camera that we possibly can,” says Heymann – and to do what feels right for each scene.”
Stranger Things combined practical effects with blue, green and LED screens, using the latter for image-based lighting and vehicle work. “Traditionally, you’d either use a process trailer or bluescreens. We started moving more of that work to LED walls and doing in-camera VFX,” Heymann describes. “In Episode 504, Steve drives his BMW through a rift in the Rightside Up, and they go out the other side in the Upside Down. Everything that was done in the car was shot with modular LED screens. “It helped the actors to know where they were,” Heymann continues. “They could see the red rift approaching and time the transition.” While the walls provided interactive lighting, “we supplemented them with LED film lights that had more punch.”
While Heymann considered using an LED volume to film the “Abyss,” an alternate dimension that appears only in the series finale, the ensemble cast threw a wrench in that plan. “Even in the biggest volume in Atlanta, at Trilith, it was going to be overly tedious. There was going to be too much walk-and-talk and too much exploration of the environment,” he explains. “I talked with David Klein, one of the DPs on The Mandalorian, who said that one of the difficult things with the volume is its size. Ultimately, we found a huge quarry that gave us a much bigger playground.”
Virtual production still has its limitations, and most innovations come through trial and error. On Percy Jackson, for instance, Gill tried to blend live motion capture with an LED volume. “There was a guy running around on stilts, and we could see him as the minotaur in the camera, but there was a slight delay. It was almost perfect,” he admits, “but not quite.”
Gill has been around long enough to see “technology start to merge with filmmaking. Percy Jackson was the first time I used the volume,” he recalls. “That was a complex piece of technology; you have to learn it, and then you have to learn how to work with VFX. You have to be the bridge between production, design and effects,” Gill shares. This is the true task of the hybrid DP.
Many cinematographers – even those on sci-fi and fantasy titles – maintain that the camera department should be kept separate from VFX. “Nobody is looking for me to do those people’s jobs, but they are needing to know that I can collaborate with those people,” Goldschmidt suggests. “It has dawned on me that if I knew how to do anything in Unreal myself, then I could cut out some middlemen, but the learning curve is fairly significant.”
“Traditionally, you’d either use a process trailer or bluescreens. We started moving more of that work to LED walls and doing in-camera VFX. In [Stranger Things] Episode 504, Steve drives his BMW through a rift in the Rightside Up, and they go out the other side in the Upside Down. Everything that was done in the car was shot with modular LED screens. It helped the actors to know where they were. They could see the red rift approaching and time the transition.”
—Caleb Heymann, Cinematographer, Stranger Things
Across the board, our DPs deny being quizzed on virtual production during job interviews. “In my experience, [directors] want you first because of your eye, rather than if you’ve worked on a volume or not,” describes Pedro Gómez Millán, DP on Daredevil: Born Again and The Lincoln Lawyer. “If you’re proficient in technology, even if you haven’t done a volume-heavy job, you’ll be able to adapt quickly,” he believes – a soft skill that goes further than any niche technical expertise.
Soffer argues that virtual technology can be “prohibitively expensive,” leaving DPs on low-budget productions out of the loop. But tools beyond LED volumes and Unreal Engine do exist, such as mobile applications Previs Pro and Polycam. Millán uses both, turning his smartphone into a virtual camera and taking three-dimensional scans of his surroundings – all without breaking the bank.
Those lucky enough to work with high-end equipment recognize how invaluable that experience can be. “It’s exciting to be working in a time when we have this software available to us,” Heymann says. “I want to dig more into Unreal Engine and get more comfortable with all you can do in there. It’s so important that as a DP, you can have a seat at the table.”
Despite the steep learning curve, there’s an undeniable curiosity among today’s cinematographers. “My journey is not finished,” states Gill, who treats every job as a chance to grow – yet knows when to leave it to the experts. “I consider myself a cinematographer, not a technician.” Instead of trying to master every tool out there, “I hire the best people.”
Virtual production was designed to solve a specific problem; it places the actor (and the camera) into any location, real or not, and offers interactive lighting – something blue and greenscreens can’t do. “It’s one more tool we can add to our arsenal,” Heymann argues. “That doesn’t mean it’s going to make sense for the majority of productions.” Goldschmidt agrees. “It’s basically rear projection in the modern era,” she says. “It’s a great tool, but during the pandemic, it was like, ‘This is how we’re going to make films from now on.’ It seemed like location shooting was going to be a thing of the past. Now that it’s over, people are dying to go back on location – and why wouldn’t you? The element of unpredictability is part of the fun.”
No matter the tool – LED volume, Unreal Engine, even artificial intelligence – the cinematographer’s job remains the same. “The technology can be overwhelming sometimes,” Millán observes, “but as the industry changes, it’s important to keep the essence of filmmaking alive.”
