By TREVOR HOGG
By TREVOR HOGG
Nowadays, real-time filmmaking involves everything from pitchvis to final pixel rendering, with The Mandalorian and 1899 showing how it can be used to achieve the best results. Portability, flexibility and affordability are areas being addressed, and, in doing so, have enabled indie projects to take advantage of the tools, technology and methodologies associated with game engines and virtual production. However, the real-time renaissance is not only occurring in film and television; it has also impacted immersive experiences, concerts and sports broadcasts. To figure out where the future lies, one has to revisit the past, starting with The Mandalorian, where multiple environments had to be created within a limited space and resources and the protagonist was essentially like a chrome ball. Interestingly, the most critical component had nothing to do with technology.
“From a technology standpoint, we knew that the pieces existed,” states Richard Bluff, Visual Effects Supervisor at ILM. “What we all needed was a goal. Whenever we walked on set on Season 1, you could ask me, Greig Fraser [Cinematographer] or Andrew Jones [Production Designer] exactly what it is that we’re trying to execute on this day, and what we were going to get out of using this technology in front of this camera. Every single one of us would say exactly the same thing. It was a massive technology undertaking, but it was a bigger undertaking of communication and understanding between departments, companies and personalities that helped it work.”
Machine learning will have a massive impact. “It’ll be another layer on top,” Bluff notes. “If a director has a casting call and has several people come in to try perform a character or a creature that’s going to be CG ultimately, and they’re still figuring out designs, the idea that you could, in theory, have somebody in real-time walking through a space, be reskinned as a CG character and, on top of that, in real-time through machine learning drive different styles, looks and weights, puts more choices back in the filmmakers hands earlier in production – that’s where you want to be so they’re more confident going into the show.”
Conceptually, 1899 still holds up. “The goal is always to get as much in-camera for virtual production as you can,” remarks Christian Kaestner, Visual Effects Supervisor at Framestore. “The dining room was probably the one set that I was most surprised by how much we got in-camera and how well it worked. Because you go in and say, ‘Oh, the sky and ocean and that all works.’ There was probably more ocean replacement than we had expected, just for the level of detail. At the same time, we gained a lot of shots in the dining room because we were much tighter and had much more out-of-focus background. That I thought was incredible.” Static skies were brought to life by adding moving clouds from Unreal Engine on top of the HDRI plate photography. “It’s easy to under-estimate how much subtlety there is in the sky movement, and the audience would perceive that.” The vast ocean was hard. Kaestner notes, “We managed to get a fair chunk of medium close-ups and defocused water. It was just the right amount of movement and white-water in-camera. But as soon as you hit wide angle, which had a large and long depth of field, the level of detail wasn’t quite there. There’s a complexity to the water movement and surface detail that is probably still challenging.”
In 2014, after working on Interstellar with his brother, Christopher Nolan, Jonathan Nolan was determined to figure out how to use projectors and build backgrounds in game engines for Westworld. Tests were conducted, but the goal remained elusive until a partnership with Magnopus was established. “We figured out what sets we wanted to use and were selective on Season 1 of Fallout,” explains Jay Worth, Visual Effects Supervisor on Westworld and Fallout. “The Vault entrance was by far the most successful set, and the farm, but we didn’t push the technology. While we were doing that, we were working on the final season of Westworld, and we started to play with the idea of, ‘Could we use the assets twice with two different environments that we needed to dress differently?’ The biggest thing we wanted to challenge for Fallout Season 2 that Jonah put on our plates was, ‘Can we start working more with natural light?’ We took our plates from Namibia, where Lucy is exiting the Vault, played with the direct sunlight, and asked ourselves, ‘Do the shadows look right? Does the light temperature look right? Can we get that feeling of if we have our plate and sun, pop out the panels and blast light onto our actors, will it contaminate the set too much, or will it look real?’”
Worth continues, “When we were up on the deck of the Caswennan, we had a nine-camera array out on Dumont Dunes shooting these drifting plates out in the middle of this scorching hot desert. We popped the panels out, blasted our actors with production lights, made sure the temperature felt right, and all we ended up having to do was clean up the panels. The team at Magnopus did a masterful job. I could show you some tests. It was literally here’s our set and digital asset, and they married perfectly in terms of where the decks begin and where the big airship starts and ends. That was one thing we wanted to try to learn from, and push the technology to figure out how we could use the tool in a way that was different from how we had used it in the past.”
Virtual production covers a wide variety of things. “House of the Dragon is not traditional in-camera visual effects via LED,” notes Christopher Cox, Head of PXO Clara (a Sony division folding into the wider Sony ecosystem). “What House of the Dragon did was leverage virtual production for real-time lighting, and then you have driving work with other projects that we do, which are either 3D or 2D driving plates leveraging gsplat, depending on the need. The technology is constantly improving every year with better panels and processing, but the biggest evolution is delivering the agility required for projects.” Technology should not dictate creativity. “It’s our job to find a way to fit within the creative needs of that project and deliver something cost-effective and efficient. When you build and spend all this money on one thing and think, ‘We must use it because we spend the money,’ that’s not healthy for filmmakers and for the bottom line on a budget. Agile and modular solutions are the best use case of virtual production. I still like the large models for specific projects. For example, last year, Clara operated on four continents at once because we were sending modular kits to different stages. They don’t always need to be at soundstages either. We can deploy panels on any film set. It’s essentially a traveling road show of workstations, processors and LED systems that go up and down quickly. Therefore, you can actually schedule much faster because we’re not shutting down your set to build this massive horseshoe. Modularity is imperative.”
Real-time technology has long been used to produce content for live events and broadcasting. “There was a really funny moment at Coachella last year,” recalls JT Rooney, Partner, Screen Producer and Creative Producer at Silent Partners Studio. “We were doing a performance for the artist, Lisa, and we were getting some technical support from the team at Epic Games. Then we realized the problem wasn’t the software. We were in the middle of the desert at two in the morning, and there was so much sand and dust that it was affecting the machines. The interesting thing about live entertainment, using the same sort of immersive technology as film or games, is that just like in film, you have lighting, set decoration, actors and performers interacting with it, but in a live environment that will change. It will float, and you have to react to it, which is why real-time has been part of live entertainment for a long time. It’s a necessity.”
Rooney adds, “When you think about audio-reactive performances or things that work with ballet or opera or dancers, that’s something which only happens once and in that moment. You need technology to update it in real-time instead of, ‘Let’s do that take again and move your hand this way.’ It’s a long discipline in that space, which is why it made it easy during COVID and the virtual production boom for a lot of live-entertainment people to get into the film industry, but bring some of that discipline in.”
“The goal of it, like visual effects or editorial, is to be invisible. There are a lot of things that have succeeded in virtual production, especially with LED volumes, that it’s discernible where you can’t tell the difference between which shot is on volume and which shot is practically done, if done well.”
—Jim Geduldick, Visual Effects Supervisor, Virtual Production Supervisor & Cinematographer
Getting projects greenlit has gotten easier. “Part of the process of even getting the commercial is a director building a pitch deck,” states Christopher Probst, ASC, Co-Founder and Chief Innovation Officer of Synapse Virtual Production. “One of our directing partners, Rich Lee, is actually using the gaming engine to build the deck, so he’s constructing environments, assets and mood boards from a 3D asset. Rich will narrate the Zoom call [with the client] while operating the environment in Unreal Engine. They’ll go, ‘We were imagining it would be nighttime and we’d be facing the other direction.’ He’ll make it night, turn the camera around and say, ‘Oh, you mean something like that?’ Rich can change the focal length of the lens. Now you can shoot those storyboards and lens choices and create an animatic, previs or techvis built on assets created for the pitch deck. If that particular production has a virtual production component, that environment could be plussed up with textures and lighting and become the asset put up on the wall alongside the live-action photography. On top of that, it’s a 3D environment.”
Probst illustrates. “We did a car commercial and created five miles of coastal highway that was bespoke to the needs of the client. They go, ‘This looks incredible. I wish we had an aerial.’ We had the 3D model of the vehicle and could drive it on our asset and program an aerial camera view. Now it’s also in post-production. One level of effort in terms of creating this environment for the pitch can carry all the way through post-production. That is a huge benefit in terms of communicating the potential of this process.”
The gap between high-end physical cinematography and real-time engine performance has not been entirely removed. “There are pain points with any of the traditional digital content creation tools,” remarks Jim Geduldick, Visual Effects Supervisor, Virtual Production Supervisor and Cinematographer. “Your Mayas, Nukes and Blenders have added virtual production tooling into them or virtual production specific workflows. The core focus of Epic Games, Unity and other real-time engines is to make games, and they’ve had to adapt to us in filmmaking. Kim Libreri has brought in other knowledgeable teams to work with the internal Epic Games teams to create filmmaker-centric tools. This has eased some of those pain points.”
Geduldick concludes, “No tool is going to be the one ring to rule them all. It comes down to the right tools for the job at hand. Sometimes you’re going to pick Epic Games and their tools with Unreal Engine. Sometimes you might go, ‘I’ve got more of a car process, plate workflow, image-based lighting.’ You might go totally media server with Nuke Stage, Pixera, disguise or Chaos Arena. You’ve got an à la carte menu these days in virtual production.” Virtual production has a visual language. “But the goal of it, like visual effects or editorial, is to be invisible. There are a lot of things that have succeeded in virtual production, especially with LED volumes, that it’s discernible where you can’t tell the difference between which shot is on volume and which shot is practically done, if done well.”
