By TREVOR HOGG
By TREVOR HOGG
There is a belief that technology has not gone mainstream until it can be found on a smartphone. LiDAR achieved this when Apple introduced the iPhone 12 Pro and iPhone 12 Pro Max. The introduction of Light Detection and Ranging is not surprising given that the camera is the cornerstone feature driving consumer interest in the mobile device. LiDAR has had a transformative impact on various industries, including visual effects. Companies like Direct Dimensions provide advanced 3D scanning and convert raw 3D ‘point cloud’ data into precise high-resolution CAD models for military and commercial vehicles. At the same time, Virtual Production Partners and Clear Angle Studios have developed a reputation for scanning film and television sets, locations and props to assist with the post-production process. The Third Floor has elevated the visualization process with LiDAR to ensure that costly creative and technical mistakes are avoided. Leica Geosystems, who manufactures LiDAR scanners, has a range of product lines that reflect the diverse applications for the technology. Finally, clients like visual effects supervisors view LiDAR as part of their toolkit to ensure that digital effects seamlessly integrate with practical elements to create a believable image.
Approximately 25 years ago, LiDAR was developed for industrial purposes. “Gas and oil plants and facilities, like offshore oil rigs, are complex, gigantic, magnificent places, and they’re constantly doing re-engineering and retrofits,” states Michael Raphael, Founder & CEO of Direct Dimensions. “The idea of having 3D data is so valuable to them. As gas and oil companies got their first whiff of LiDAR, they would underwrite R&D companies to make it smaller, lighter and faster, which drove its development. Eventually, LiDAR got picked up by surveyors responsible for measuring terrain and buildings for construction. Probably eight or 10 years ago, a little bit got picked up in the autonomous vehicle world and automation in general. Today, it has gone into all kinds of things including film production. Around 80% of the market is Architecture, Engineering and Construction (AEC). The idea of scanning ‘the as-built world’ is gigantic. If you are an architect hired to renovate a building, you would go out and do a LiDAR scan to create an ‘as-built model’ so you can do the renovation design in the computer. The idea of construction scanning is even bigger than that in the building space. Construction companies scan every couple of days or weeks as they’re building, so you can document where everything is inside the walls and building. If you scan the building as it’s being built and put all of the data together at the end of the day, when that building is completed it’s like having a CT scan of a human body.” Diversity rather than specialization reigns supreme at Direct Dimensions. “We do many different things, but fundamentally we scan everything. Every day, we go in six different directions, and it’s hard, but it has kept us able to ebb and flow as the economy moves in various directions as it does.”
LiDAR scanners have entered the realm of portability. “The original scanner was this massive box that would sit on a big tripod, it had to be connected to your computer and plugged into a processing unit that had cables which had to be lined up in a big circle,” remarks Andy Fontana, Product Marketing at Leica Geosystems. “That was in the early 2000s. Now we have a LiDAR scanner that can do much more and is the size of a can of Coke. A tremendous amount of innovation and technology has been squeezed into a smaller package. Also, the price and accessibility to the technology have come down, so it has opened up the market to what used to be a dedicated customer to this technology to something that a prosumer can use in their work.” There are different levels of LiDAR scanners. “Manufacturing is doing it on a level of microns, and that’s a level of precision that isn’t even remotely necessary for most of the other industries in the world. There are people who are scanning bridges or a massive infrastructure project and need a LiDAR scanner with a range of half a mile and accuracy less than a millimeter. Those scanners are $150,000 to $200,000, big, bulky and made for construction sites. On the lower side, you have something like our BLK scanner, which you would use for a scout, which is more in the $20,000 range. The range is about 30 to 50 feet. You’re scanning an interior or smaller exterior space or object and still get a great idea of that environment. You don’t need such a powerful LIDAR, which brings the cost down, makes all of the components a lot smaller and turns it into something that you can throw in a backpack.” The construction of the device has been altered. Fontana notes, “Traditionally, the LiDAR has had numerous pieces, which are subject to damage. What we are doing with our new scanners is making that LiDAR solid state, so there are no moving pieces. It’s a lot more durable and will last a lot longer.”
Virtual production has caused LiDAR scanning to become an essential tool in creating virtual versions of actual locations. “Productions can now capture detailed LiDAR and photographic data to generate photorealistic digital environments,” states Dominic Ridley, Director & Co-Founder of Clear Angle Studios. “This allows directors and teams to explore virtual locations in VR before the actual shoot takes place, offering a new level of flexibility and planning. This is especially useful for films that need complex set extensions or large-scale environments, where building everything physically would be impractical or impossible. By using LiDAR for virtual scouting, filmmakers can make better decisions about camera placement, lighting and scene design, saving time and resources during production. The ability to navigate these virtual environments also helps teams visualize and adjust scenes long before they hit the physical set.” Limitations do exist when it comes to color and texture resolution. Ridley observes, “This can hinder its use in applications where visual details are crucial, like in detailed architectural renderings or film production. Depending on the equipment, the resolution and point density can also be an issue, especially with mobile LiDAR systems where you often sacrifice level of detail for mobility. Another challenge is that the need for higher scanning resolution can slow down data collection or require more advanced [and expensive] equipment. Since LiDAR can generate so much data, managing and processing it can be computationally intensive. For industries like visual effects or autonomous vehicles, this can slow down real-time processing, and the large file sizes create storage and data management challenges.”
Smartphones with LiDAR come in handy. “If we are ever in a pinch, and it’s a scene that I didn’t expect anything to be needed, and suddenly we have to add x, y and z, rather than bring out a LiDAR team, I do a quick scan on my phone,” remarks Mark Russell, director and VFX Supervisor. “At least it gets you the basic geometry of a scene, which greatly helps in recreating stuff. There is an app called Polycam that is spectacular when it comes to that.” Cloud services assist with processing power, sharing and portability. “On the last show I did, we staged everything in the cloud, and we were able to use that to help distribute the material and re-mesh some of the point clouds. But it all could be done on a local machine. It’s just easier when you do it in the cloud. You don’t have to carry the thing with you. Cloud computing is changing and shaping things by using photogrammetry as a way to supplement LiDAR or possibly replace it. You have processor-in-tense algorithms to figure out the geometry of that scene from 50 pictures. LiDAR is still reliable, fast and more accurate; however, in a pinch it’s the other stuff that takes more advantage from the cloud computing.” The desire is for the highest quality imagery. “It’s always about capturing the most data you possibly can, then figure out what you need later. The idea now is that you can use LiDAR scans and bring your point cloud into your camera-tracking software, which will help make the camera move better in the computer. The better data you get, the better the results are down the road.” LiDAR is getting to be more universal in the film production process. “Originally, only a handful of companies knew what to do with the data from a full high-end LiDAR scan. Now, everybody knows what to do with it. Art departments often use LiDAR to build their sets or design a location where we’re going to shoot. If we know that we’re going to shoot in this location, I will always try to get in early because everyone along the way can use that data, which is great.”
Some of the applications for LiDAR have been surprising. “A game-changer for me that I didn’t initially realize was possible was extracting single points from within a Leica Cyclone Register 360 Plus,” reveals Casey Schatz, Head of Virtual Production at The Third Floor. “LiDAR will not pick up gaff tape marks or Sharpie marks on the ground. Yet, these markings are usually how you lay out critical construction, set decoration measurements and known points for your motion capture volume. Before using the scanner, I figured I’d have to have the corner of an apple box at the marks or some other calibration object that would be picked up in the scan. In Register 360, you can be inside the panoramic sphere of your scan location with the actual photo and pick points at the middle of that Sharpie mark with millimeter precision. These coordinates output to a text file that is easy to import as locators in your 3D software. This is how we have done all our live-action mocap volume offsets to the digital environments. Another cool thing we stumbled upon was other departments, particularly lighting and rigging grips, benefiting from quickly being able to look at the set from places you normally need a cherry picker to get to. As I mentioned earlier with the 360 scanning, you get everything, which means you’re also getting the catwalk, truss, chains, forklifts, and anything that makes a movie set function. More and more, as live-action progressed, the rigging folks look over my shoulder, and we’d do some live techvis just utilizing the colored point cloud and measure tools within Register360.” LiDAR has been invaluable while filming the Avatar sequels for the eyeline system and Simulcam. Schatz says, “The eyeline system is a small monitor and speaker that moved around the set via four cables, much like a cable cam you’d see during a sporting event. It was a neat way of physically representing the CG character on set for the live-action talent to respond to. As the set was being lit and other departments would descend onto the set with their required gear, I would do a LiDAR scan almost hourly and run the simulation to ensure the cables would not intersect. This let us find clashes before running the system and allowed us to make slight adjustments and compromises.”
Digital replicas of distant locations have been created for studio projects so production crew members can access them remotely. “We have also done digital replicas for fine artists and other folks who are starting to see the value in different situations,” notes Conrad Curtis, Co-Founder of Virtual Production Partners and a 2nd assistant director. “This has been normalized in the visual effects side of our industry, and other industries are starting to realize that it’s more valuable and more acceptable than they might have thought before. One of my favorite quotes from Jensen Huang, CEO and Co-Founder of Nvidia, is, ‘Digital twins help companies build very complicated things perfectly the first time.’ He’s not talking about entertainment because if he were, then the entertainment industry would be much more functional. Jensen is talking about heavy industry and manufacturing. Digital replicas are beneficial because people can look and talk about something, explore a space or idea before they show up. Nothing is worse than being on set and having this time where people are figuring out why everything is not working. Now we can allow a director and some actors to take a rehearsal, use virtual production in that rehearsal, and have an edited version of their movie before they get to the set. I’ve been working with a director who is able to understand what lines and shots to throw away because he’s already done a cut of the movie inside of a digital replica. That’s the kind of power people don’t quite understand. Coming from the perspective of planning people, that’s where we get to see these technologies blossom into a new space.”
