Virtual reality and augmented reality are in their embryonic stages but are predicted to become a $150 billion business by 2020, surpassing the mobile industry. That matters to the visual effects world, because its artists’ skills are a perfect match for what’s required to make convincing VR and AR.
Already, nearly every media and entertainment company from Pixar Studios to IMAX is creating virtual reality content. Studios have created VR experiences to accompany movies as disparate as Wild and The Martian, and director Jon Favreau just created Gnomes and Goblins, an interactive VR experience that pushes the boundaries of what the medium can do. Dozens of VFX artists are gravitating to the new medium.
At USC’s Entertainment Technology Center, VR/AR initiative project lead/consultant Phil Lelyveld – who notes that the new media will create “whole new markets” including training, customer support and marketing – differentiates virtual reality from augmented reality. “If you’re walking down a supermarket aisle, holding up your cell phone and the camera gives you information about the products in front of you, that’s augmented reality,” he says. “If you’re at home in an easy chair, directing a robot down that supermarket aisle, all the sensory input is coming from a place you aren’t actually in. That’s virtual reality.”
Mixed reality describes the continuum between those two extremes, based on a scale established during virtual reality’s first boom in the 1990s. “As you increasingly add virtual things, it flips from augmented to virtual reality,” explains Lelyveld. “VR glasses isolate you, and AR lets you see the world around you, but in both cases, you’re inputting virtual sights and sounds.”
The barrier to entering the VR/AR creation community is not high for VFX artists who are already adept at creating and manipulating computer-generated imagery. Unity and Unreal are the two game engines on which VR experiences are based, so that’s one new skill some VFX artists will need. “But once you learn those, there’s a big need for visual effects artists to make VR a quality experience,” says Lelyveld. Another added value that visual effects artists bring to the table is expertise with 3D stereoscopic imagery, honed during the push for 3D in movies and TV.
The VES Vision Committee, chaired by Toni Pace, has held events on virtual reality, including “Storytelling in VR” and “VR Post Production.” Scott Squires, VES, a member of the committee who has played a leading role in its VR initiative, is an example how veteran visual effects professionals are repurposing their VFX skills for the new medium. As Chief Technology Officer and Co-founder of virtual reality company Pixvana, Squires has directed or supervised numerous 360-degree videos for viewing on VR headsets, built and programmed specialized video players for VR headsets as well as a number of interactive 3D experiences for VR headsets using Unity and Unreal.
Squires explains how he became interested in virtual reality. “With the advent of the Oculus DK1 [headset] it seemed VR had real potential at a near point in the future,” he says. “With the arrival of the Oculus DK2 and certainly the prototypes of the HTC Valve Vive, the quality and experience levels raised enough to see the potential of VR.”
Squires points out that VFX is a great background for a number of areas of VR. “Anyone on the 3D rendering side – modelers, painters, lighters, animators, particle artists, developers and so on – can apply much of what they’ve done to make 3D worlds for VR,” he says. “It’s similar to the transition from visual effects for film and TV to those for video game production. Typically, VR production means creating assets for a game engine, with further manipulation in the game engine.”
The expertise required for VFX that can be applied to VR extend beyond the creation of 3D assets. “The skillsets of personnel involved in post production, including compositors, rotoscopers and so on, apply directly to stitching and blending 360-degree videos,” Squires says. “This is a much needed area to achieve the best quality. Nuke, After Effects and specialized tools are used for this type of work.”
VFX professionals on the live-action production side also possess skills important to virtual reality production, says Squires. “They understand the need to shoot, mindful of the requirements of post production,” he says. “Issues they would be sensitive to include stitch edges, which can be thought of as matte edges, camera calibration and dealing with multiple camera alignment. Volumetric capture takes many of the photogrammetry or Lidar VFX skillsets and applies them directly to build VR worlds.”
Squires points out that “another advantage for VFX artists is that we tend to understand the 3D world and can imagine working in that space.” Likewise, he says, “there are some VR tools that could be a real boon to VFX.
“VR provides the ability to see a model spaceship or set in real scale,” he says. “It allows you to walk around the previs design or set designs before that work is done. There are tools to paint and model in 3D while in the headset, and that’s a much different experience than looking at a monitor and using a tablet.”
“Anyone on the 3D rendering side – modelers, painters, lighters, animators, particle artists, developers and so on – can apply much of what they’ve done to make 3D worlds for VR. It’s similar to the transition from visual effects for film and TV to those for video game production. Typically, VR production means creating assets for a game engine, with further manipulation in the game engine.”
—Scott Squires, VES, Chief Technology Officer, Pixvana
SpinVFX in Toronto is among the visual effects facilities that are now using their knowledge and experience for virtual reality projects. “After 30 years in the film and television industry, it is no wonder that we are pushing the envelope and looking at what’s next,” says President/Executive Producer Neishaw Ali. “Virtual reality is such an immersive experience and is a natural fit for SpinVFX, as we are a content provider.” She enthuses over the new medium that “allows you to be physically transported” and “interact within your environment. It’s exciting, unnerving and challenging,” she says.
Ali reports that her company’s experience in creating photoreal imagery has been applicable to its new focus on VR. “We know where the uncanny valley lies,” she says. “We know how to play to the medium’s strengths and adjust to accommodate for what the naked eye needs to perceive.” Because VR is a new medium, she adds, “there are plenty of challenges to contend with.”
Those challenges have been both technological and creative. “From the technology side, we are learning a lot about real-time rendering from video game engines,” she says. “We’re also learning about the importance of scene optimization to maintain frame rate.” From the creative side, she says, the company has learned that planning for the user experience is critical. “The relationship between the inner ear and our eyes is a delicate balancing act,” she says. “We always need to stay within a certain level of tolerance that doesn’t run counter to human physiology. We often act as a guide to maximizing the experience with interesting things to look at, listen to and interact with, so the one thing flows into the next.”
To target new markets for VR and AR, including architectural visualization, healthcare, automotive, product design, advertising, social media and games, SpinVFX has developed BRIO VR, a platform to create 3D virtual reality experiences.“The platform is designed to be as simple yet powerful enough for anyone to use, without needing to code,” says Ali, who reports that the platform has a template library for auto design, VR photo/video galleries, architectural visualization and others. The platform connects to HTC Vive, Oculus Rift and Gear VR headsets. A completed design can be shared via a link.
VES member Jacki Morie, who helped found USC’s Institute for Creative Technologies (ICT) and who wrote her 2007 Ph.D. dissertation on VR as a new medium, started her own company, All These Worlds, to focus on virtual world projects. Her company’s work also provides ample evidence of the variety of projects that can and do employ virtual and augmented reality. All These Worlds worked with NASA to create a virtual world ecosystem to mimic Mars, which was installed in a geodesic dome in Hawaii. There, scientists were sequestered for a full year to test the impact of social isolation and sensory deprivation. At the end of the study, the scientists that could interact with the virtual world showed better psychological health than a control group that didn’t have the system. Morie presented a workshop at NASA on the results.
She reports that many rank-and-file visual effects artists who have worked in all aspects of film VFX are getting involved in VR to expand their opportunities and skillsets. “There’s a need for modelers, for people who can do environment design and look – all the things we do in VFX,” Morie says. “There are beautiful visual effects that would translate beautifully into VR. The things that grab our attention for film also work in VR.” The differences that VFX artists have to learn, she says, is that the environment is totally immersive and gives agency to the viewers to craft their own stories.
“The message about VR that I’ve been trying to get out since I became involved in 1989 is that VR is a totally different medium,” Morie says. “We’re at that moment with VR that we were with film at the turn of the 20th century. But some of the best VR has beautiful effects with moody lighting. That’s what VFX people can bring to VR: the artistic sensibility and the production expertise to bring it into reality.”
“The skillsets of personnel involved in post production, including compositors, rotoscopers and so on, apply directly to stitching and blending 360-degree videos. This is a much needed area to achieve the best quality. Nuke, After Effects and specialized tools are used for this type of work.”
—Scott Squires, VES