UNSUNG HEROES OF THE PIPELINE: ENCODING ARTISTIC VISIONS

By TREVOR HOGG

The visual effects pipeline aims to allow artists to focus on creative challenges rather than technical ones. However, technicians must continually update the technology to ensure the best tools are used, and that implementation doesn’t cause costly production delays. The pipeline department comprises developers, engineers, leads, architects, technical directors, and supervisors from various career backgrounds. The major goal is to translate the artistic needs into programmable computer code.

“We work in the shadows, but our teams are great developers passionate about visual effects and can do a ton of magic,” states Anne-Gaëlle Jolly, Head of Technical Directors at Rodeo FX. “You have to juggle 20 things at once but always give the feeling that everything is under control. A key part is to stay calm and don’t hesitate to take a step back to make the right decision when I’m trying to ‘put out fires.’” The pressure can be intense. “We had to roll out an OS change and Python version change across the farm and the floor at the same time while having a full slate of running shows and making sure that this would not impact the artists,” remarks Julien Dubuisson, Head of Pipeline at Rodeo FX. “We work with every department from production to the artists on the floor. Some are more technical than others. We will get detailed specs on all aspects of a tool or system or a rough outline of the expected outcome. They could work without us, but the systems we provide significantly enhance their ability to work at a much greater scale and efficiency.”

“We work in the shadows, but our teams are great developers passionate about visual effects and can do a ton of magic. You have to juggle 20 things at once but always give the feeling that everything is under control. A key part is to stay calm and don’t hesitate to take a step back to make the right decision when I’m trying to ‘put out fires.’”

—Anne-Gaëlle Jolly, Head of Technical Directors, Rodeo FX

Python is a prominent pipeline programming language. “Ninety-nine per cent of our pipeline code is written in Python. The rest is configuration files and the occasional shell script,” explains Nicholas Hampshire, Lead Pipeline TD at Framestore. “That hasn’t changed in over a decade. We use Maya, Houdini, 3D Equalizer, Blender, Mari, Mush3D and other great off-the-shelf software for CGI All require deep in-house integrations and customization to ensure we can deliver the best for production. We’re picking up ML and formalizing our CI/CDs to make it easier to test our code before rolling it out.” Mental strength is important. “It’s reasonable to say that you must be dogged and have a grizzled determination to succeed. Software development is only half the problem. The rest is an ongoing battle against time!” Hampshire gives an example where he had to conquer the clock. “I was left with no alternative than to re-implement a deprecated C++ render-time point instance format within a distributed set of Python expressions embedded in a VEX SOP inside of Houdini and get it to match renderer results within three hours, one idle Thursday night for a major feature film. That was reasonably tricky to do. Worked alright, though.”

The cornerstone of the pipeline is the asset management system. “The most important thing for the studio is how you manage your assets,” remarks Elizabeth Shiers, Lead Pipeline TD at Outpost VFX. “Where is the central database you store that information? For Outpost VFX, that’s Flow for production tracking. Your DCCs need to plug into that database. It’s about how you get from that database into a DCC. Do what we need to do in the DCC and pass it back to the database. The next person who comes along pulls in again from that database to another DCC, does what they need to do and gets it back out. That process of getting it in and out is where the pipeline comes in, trying to make it seamless between DCCs. The aim is to make it feel familiar. If you’re a generalist and need to switch from Maya to Houdini to Nuke you are familiar with, you know ‘This is how I get things in and out.’ They know the DCCs enough in a vanilla capacity to do what they need to do, and the pipeline is there to pass data from the database into the DCC and then to someone else. That’s how I see it working in the ideal scenario.” Outpost VFX came into existence after most third-party standards were established. “DCCs support the things we’re trying to use. It’s a matter of getting them from one to the other.”

“The most important part of my job is leading a team to automate all of the boring stuff so that artists can focus on making art,” notes Chad Dombrova, Global Head of Pipeline at Scanline VFX. “Another significant aspect is taking all of the stories you hear from the different artists in the various departments, seeing the overall trend of these problems and finding the right technology to pair up with them. You’re trying to build a set of technological capabilities that can solve almost any problem. That’s your goal.” Reducing the time for artists to get iterative feedback has been the goal for over the past 30 years. Dombrova adds, “The bar is always being raised as we get closer to that nirvana, which makes it harder to achieve real-time. However, as a principle, the pipeline should be built in a way that technologies are ideally interchangeable. If there’s this new real-time technology and we feel it will have a huge impact, we should hopefully be able to excise the technology it replaces and slot it in. One of the things that machine learning is good at is simply taking a process and making it more efficient, like de-noising. As a pipeline engineer or architect, what you looking is choosing an ecosystem and investing in those types of technologies because, often, the easiest way to get those kinds of technologies into the pipeline is not by making some massive switch. It’s by adopting a DCC software vendor that is already investing in those things for you.”

“There is this sense that if you are entrenched using a particular software then moving away from it, you need to weigh the cost-benefit of introducing something completely new or a change to the workflow,” states Claude Martins, Director of Pipeline, VFX at Digital Domain. “About five or six years ago, we switched from using Maya as our primary lighting platform to Houdini. Houdini had just come out with their lighting platform Solaris, which was USD-based, and what we saw there were the advantages of getting our foot in the door for introducing USD into our pipeline, which more and more of the industry is switching to. And the ability to switch to sequence-based workflows. Before, we would have lighters light each individual shot. We were able to introduce this huge efficiency to lighters. While it was quite painful in terms of the effort to switch over, once we switched over we immediately started seeing the benefits. The other example is our crowd soft-ware. Despite investing a bunch of effort into our Houdini Crowds pipeline, which is simulation-centric, we looked at the costs and benefits. We decided to switch over to using agent-centric Golaem. It’s always a case-by-case basis.”

Generally, the fundamental tools used by Pixar have remained relatively stable. “There are moments where we have a big transition point,” states Ariela Fedorov, Technical Director at Pixar. “For Brave, we switched to our new proprietary animation software Presto, and for Finding Dory, we changed to a new version of RenderMan RIS and Katana.” Technological development is divided into three areas: R&D explores cutting-edge technology, tools take a long-term view of pipeline and software engineering, and global technology deals with technological issues and provides support for specific shows. “Soul was our first volumetric characters, and we’ve had other shows since then like Elemental. We didn’t have a way to converge geometry into a volume, so we had to make a pipeline for it; that pipeline has been developed and become a central pillar of our workflow.” Art and technology impact each other. “What is healthy for a show is to have the art influence the technology that we need to build and support, but also have the technology influence the art because we’re not working in a vacuum. There are constraints such as schedules and team size. We need to ensure that the artistic look we land on is something that can be executed and scaled.”

A specialized approach is taken at Sony Pictures Imageworks where the pipeline team supports layout and animation. At the same time, look development creates the specialized tools to achieve the desired visual aesthetic of a project. “We mostly work within Maya, at least for CG shows, while in visual effects, we have more support for the back-end department such as Katana and Nuke,” states Diana Lee, Pipeline TD at Sony Pictures Imageworks, who also has to assist in providing assets to other visual effects studios. “Especially if a rig is being shared, we have a workflow that pulls out all of our SPI plug-ins before we send it out. That file can then be opened in a vanilla version of Maya. You don’t need any plug-ins. If we’re sending just the geometry, it can be sent using alembic caches that don’t have proprietary info. We have also recently been using USD publishes to send info to other studios. It’s a challenge to keep track of everything we need to send, what we can send, and how we can send that efficiently because those requests always come in.” The really difficult part of the pipeline is understanding the issue being brought forth. “We have to be good communicators and, on top of that, be good on the technical side. The fun part of being in pipeline is being that glue between the artists and development. The artist tells you something they want, and it’s your job to clarify, ‘Do you need A, B and C?’ Sometimes, what they’re asking for is already possible, but maybe they don’t know about it, or other occasions, it’s like, ‘Wow. That’s a good idea. We’re going to put our heads together and make something out of it,’” Lee says.

“We have to be good communicators and, on top of that, be good on the technical side. The fun part of being in the pipeline is being that glue between the artists and development. The artist tells you something that they want, and it’s your job to clarify, ‘Do you need A, B and C?’ Sometimes, what they’re asking for is already possible, but maybe they don’t know about it, or other occasions, it’s like, ‘Wow. That’s a good idea. We’re going to put our heads together and make something out of it.’”

—Diana Lee, Pipeline TD, Sony Pictures Imageworks

Standardization exists within the visual effects industry. “At the moment, the file format standards are open-sourced, so all of the applications we use you can write in the same file format, meaning that the things we generate are highly portable,” observes Elliott Smith, Head of Pipeline at One of Us. “That has now been put on steroids with USD where it’s not only things we can share with other companies or people like models or renders but full complex scenes with lots of things going on.” A lot of information must be stored in a database such as ShotGrid. Remarks Smith, “What are the relevant details about that thing the next person needs to know? Preserving that information is important because we can only automate if we’ve got the right input and what we’re trying to output. How do we move things through the pipeline and make it all work? We have to concentrate on, ‘Is the thing that we made technically correct?’ There is a big focus on validation checks. We do need someone to look at it in dailies and say, ‘Yes. Ready for the next person.’” An overall view of the workflow has to be taken into consideration when devising solutions. “If you’re making something for mass use, you have to go through all the departments and ask, ‘What do you need?’ You can’t assume that everyone needs the same thing. We don’t want to bias the pipeline to one particular individual, shot or show. We rely a lot on our HODs and supervisors on shows to guide us,” Smith adds.

Even though there are a lot of crossovers between animation and visual effects, pipeline differences do exist. “All of the major tools are the same,” notes Larry Gritz, Software Architect & Sony Distinguished Engineer at Sony Pictures Imageworks. “There are stages of the pipeline that are more relevant to one than the other. Digital doubles or face replacements are not required for animated shows. The biggest difference these days in many ways is that on the animated shows we do the whole thing, whereas for the live-action effects, almost everything is split between four or five major studios.” Each department picks a central software program based on what works best for them. Says Gritz, “Pipeline is concerned with figuring out how things will flow through these different applications that come from several different companies. The custom software is written either by the pipeline or software group. When I started, every DCC that existed was its own universe, and every studio was a silent thing with a bunch of custom development that was nothing like what you find in a different studio, even though they were working on similar things. Now, there’s a lot more commonality, and part of this is the commonality of the foundational software that we’re all using and working on together. Also on the visual effects side, the need to work with the other studios on the same shows and be able to exchange assets and deliver uniform results to clients is driving more of a standardization.”

Complex tasks revolve around cross-departmental workflows. “No one works in isolation, so any change in process can have significant impacts both upstream and downstream,” remarks Anthony Winter, Pipeline and Software Supervisor at Rising Sun Pictures. “One area we’ve been refining is the QC workflow. The goal is to ensure that each department’s output meets the needs of all subsequent teams while managing the smooth flow of assets through the pipeline. This involves determining how to present the right visual representations to the right people at the right time, gaining approvals efficiently and quickly moving assets forward in the pipeline. Technology alone cannot address all these challenges. Striking the right balance between human tasks, metadata and automation is incredibly complex, especially when accounting for the unique requirements and priorities of different shows. Improvements are ongoing, and there’s still much to be done to optimize this process.” There have been technical highlights. Recalls Winter, “A recent proud moment came when we successfully scaled our render farm to nearly 10 times its compute capacity, leveraging both on-premises hardware and multiple cloud-based compute facilities. This achievement wasn’t just about increasing compute power. It meant handling 10 times the jobs, 10 times the data and countless high-priority requests. Essentially, everything ramped up to 11, and we managed to keep it all running seamlessly.”

Each project for Cinesite is given a separate pipeline, though there are occasions when the same one is used. “We use something called Git, which is the backbone for all of the software,” states Sherwin Shahidi, Head of Production Technology at Cinesite. “If overnight one of these pipelines breaks, we have GitLab, which allows us to place a temporary pipeline right away that doesn’t have the problem so the project can continue. We have a timestamp of all the steps that we have done.” While visual effects and animation utilize the same pipeline, there are quite a lot of differences. Feature animation requires better rigging and blend shapes. The lighting depends on the vision of the art director and director. We have to go back and forth quite a lot versus visual effects, where some shots are full CG, but for the majority of them, we need to match something that is in real life. Cinesite made a conscious decision to use the same pipeline, and we make customizations at the department level for what they need. All of our pipeline engineers work on the same pipeline, whether it’s feature animation or visual effects. If you look at the water in Moana, it is exemplary and uses visual effects. Over the past 10 years, there are quite a lot of shared elements. Last year, we started a visualization division, which is techvis, previs and postvis. The idea is to put together a vision of the director as quickly as possible into a workable visualization, and we use Maya and Unreal Engine for that. The turnaround to get something is usually a maximum of four weeks. That’s where real-time has helped.”

Whiskytree has a mixture of generalists and specialized digital artists. “In our case, it has worked out well where the 3D artists are generalists and can dabble in a lot of different things,” explains JP Monroy, CG Supervisor and Director of Artist Development at Whiskytree. “But there are occasions when you need someone who is so specialized and focused on one tool that they’re the only ones who can work on it, so you need to cater to them.” Pipeline alterations are handled delicately. “Updates are done by our tool developers in the middle of the night or when fewer people are working. That way, it’s the least disruptive. Software is getting updated all the time. Most studios and people want to stay on a version to complete a project, so you can take the time to update and test out new things during that in-between period.” For the past three decades, the core computer programs in the visual effects industry have remained the same. Observes Monroy, “Maya, Nuke and Houdini stick around because they’ve been the basis for a lot of creative work, and a lot of studios have built a pipeline and workflow around them. Changing all of that at once is like trying to turn a tanker. It will take way too long and way too many resources. However, other studios coming up with people who have no experience in that, then they might start to try new things and make less use of some of those tools. For now, it seems they’re the standards, and it’s how many of us are used to working, but things are constantly changing. Who knows what five years from now will look like!”