This is our technical look at Toy Story 4. Also, check out our podcast discussion on the VFXShow.

Toy Story 4 (ST4) continues the saga, picking up from the end of TS3  where Woody and Buzz, along with their other toy friends, have found new appreciation after being given by Andy to Bonnie. The audience is also introduced to Forky, a spork that has been made into a toy, and Bo re-appears, as Andy’s ‘special friend’ having not been seen in ST3.

Bob Moyer was the Supervising Technical Director on Toy Story 4, which on a Pixar film such as this is the closest being the film’s VFX supervisor. The Supervising Technical Director is in charge of the film’s nine main departments (except for animation), including characters, environments, global technology, simulation, crowds, lighting, and rendering. “It’s basically all the non-animation departments – we are responsible for the visuals,” Moyer explains. “My role is both as a kind of a general contractor to figure out how we’re going to build the visuals for the film, as well as acting as a consultant for the producers and director about what’s the best way to do things, what’s feasible and what would be difficult.”

Toy Story 4 is a remarkable piece of animation, modeling, and storytelling done by a passionate team with an incredible eye for detail. Because the main characters are toys, fine details are important as they showcase the small size of the toys and their unique world perspective. For example, in these images above – shot in the carnival game booth, the blown up bottom image shows how the artists and technicians were able to really polish the set and environments. The team added details such as slight bends and welding detail in the metal grid near Buzz Lightyear. While the audience may not focus on such detail, it all contributes to the cohesive world of the characters. It also all adds to the scene and shot complexity. A complexity that Bob Moyer and his team had to manage.

One of the most exciting technical aspects of this film was the approach to lighting and in particular, the lights in the carnival environment. Pixar has two Directors of Photography on a film such as this. One is responsible for blocking and composure. The other is responsible for lighting. For this film, Patrick Lin was the first DOP and Jean-Claude Kalache was the second Dop, responsible for lighting. Kalache had previously lit UP, Coco and most recently the short Bao. “JC Kalache and his team were really concerned about coming into the carnival and having to basically rebalance the lighting with every single shot,” explains Moyer. “And they thought it was a really excellent opportunity to essentially light a set the same way you might do on a live-action set.” The lighting and sets teams literally went down to the light bulb level and agreed on what the Kelvin scale and intensity of every single light bulb was going to be in the carnival. They then instanced those blubs onto all the rides. All of the lights in the carnival and all the lamps in the Antique Mall were calibrated. This meant that the lighting process adopted a new approach when starting on any set. Where previously their first step would have been to start to drop into gaffed lights or ‘practical’ lights. Now they start by setting an exposure control inside of Katana. Since the lights were a constant that had already been decided, “the problem effectively became a modeling problem for them, they could choose whether something should look like it was daylight or nighttime by just setting that exposure,” explained Moyer. “It was just an amazingly powerful way for them to work where the set almost did the lighting for them.”

The system took some months for Moyer and his team to get working, but when it was working, lighting almost became a problem of working out what lights to turn off not adding lights. “Rather than starting a shot and turning the lights on, the lighting team would go into the Second Chance Antique Mall and decide which lamp they wanted to turn off. They would then spend their time on figuring out the cinematic approach of, say ‘Oh, we actually want a sunbeam coming exactly along here’. Or ‘we want to get a general feeling of cool or warmth in this shot’ – as opposed to spending half of their time just trying to get to the point that we could start at by default,” adds Moyer.

Both DOPs on the film were keen to explore this more physically accurate approach. Naturally, as the work was fully animated the team could also control the dynamic range. For example, Pixar was not constrained by trying to hold exposure through windows or avoid overexposure through a doorway. The team even debated if the digital camera should have the aperture change the depth of field automatically on the digital cameras. “We went back and forth on a bunch of these points…and we decided that was perhaps too powerful a thing that we didn’t want to get locked into,” explains Moyer.

David Luoh was the Sets Technical and Extension Lead and his team worked with the art and story departments to figure out what things were needed in the sets. Many of the items in the Second Chance Antiques Mall were from the Pixar asset library, but others were very film specific. For example, the shot where Bo and Woody climb up near the rafters and see the beautiful array of chandeliers highlights how things have changed from the earlier Toy Story films compared to this film. Clearly, the characters and rendering technology has evolved, but the changes are much deeper. It is almost impossible to imagine this shot done on TS1 or TS2. Apart from the vast rendering complexity, the sets team previously would have built sets and only much later these would be fully textured, lit and rendered. In contrast, the chandeliers shot needed near final rendering just to design and block. Some sets only work with the complex ray tracing that the new RIS RenderMan so seamlessly provided. Luoh points out that the process is now very nonlinear. “There is a baseline of trying to work efficiently and being as streamlined as possible, but there are situations like this chandelier shot for instance – or another one would be in the beginning when Woody and Forky come upon Bo’s lamp in the window – in both cases, there’s just a heightened sense of collaboration and iteration with the other departments,” he explains. “In those shots, it was important for set artists when they were building the sets and constructing the environments to have as close a representation of the final lighting as possible.”

For Toy Story [4??], Luoh states that his set extensions team’s “modus operandi is to do things as cheaply, with as few people and as quickly as possible.” There were many creative workflows, and technical optimizations that were put into place, especially in areas such as the carnival lights and in small details such as with the dust and cobwebs.

The artists created detailed sets and because the main characters are toys, tiny details are important, as they showcase the small size of the toys and their unique world perspective. In the images above of Second Chance Antiques, artists and technicians added a layer of dust to ‘sell’ the set as real. They were able to dial up or dial down the dust to best serve the needs of a given scene.

Another huge technical tool that increased efficiency was the heavy reliance on USD. It is possible to load the entire Second Chance Antiques Mall with all its props and the entire carnival into one USD master file and navigate from a chair to flying above the town. fxguide got a chance to play with this first hand at WWDC. Admittedly, the system was running on a new Mac Pro but the underlying USD logic is the same that Pixar uses and the assets were the production assets from the film (without the characters).

Bob Moyer explained the significance of USD and how central it now is to Pixar, saying, “I honestly don’t think we would have been able to get the film done without it. Over the years it has become more and more the central flow through our pipeline. Whether it is how we handshake from one app to the next, or how things move from our in-house software, Presto, to the Foundry’s Katana and out to Houdini – it is central. When you’re talking about sets as complicated as the Mall,” continues Moyer,  “there’s just no way I think any of that data would have made it through our pipeline without USD.”

The company uses a small piece of software called USD view which allows direct viewing of USD files. “It works like an Open GL viewer and you could actually bring up the entire carnival and the entire Antique Shop and pivot around them in real time with the flat version of the set.” A layout artist might use this to find a shot or stage the next scene with the particular lenses the DOP wanted. If the entire TS4 set was loaded and viewed the USD viewer would be accessing 41.7 billion triangles in the dataset, (although there is aggressive culling for things not seen or too small to see at any time).

USD: Universal Scene Description

USD is rapidly gaining huge adoption throughout the industry. CGI pipelines capable of generating, storing, and transmit huge quantities of 3D data. In Pixar’s conceptualization of the problem, this is all just a “scene description.” Each of many DDC (Digital Content Creation) applications such as modeling, shading, animation, lighting, fx, and rendering often have their own special formats which are neither readable nor editable by any other application. Pixar open sources USD and it was the first publicly available software that addresses the need to robustly and scalably interchange arbitrary 3D scenes that may be made from many different elemental assets.

This image shows the team exploring camera placement within the virtual set. The outward-facing goal of USD is to take the next step in DCC application data interchange beyond what is encodable in the open source Alembic interchange format. The chief component of that step is the ability to encode, interchange, and edit entire scenes with the ability to share variation and repeated/instanced scene data across shots and sequences, by providing robust asset-level file-referencing with override capabilities. As one can imagine, Pixar is advanced in its use of USD.

Above is one of the 12 sets inside the Second Chance Antique Mall. By operating with this approach, artists do not need to wait for render time and the team can access vast datasets. A previous tool at Pixar called Scene Scout also used USD, but it treated the USD more as if it was a DCC. As such it would import the USD into its data structures and thus it had a very finite amount it could access at any one time. USD view leaves the data on disc and then it effectively maps in what it needs and does not attempt to convert all the dataset to a second format. A huge aspect of why this works is instancing. For example, all the light bulbs and all the rivets in the rides are instanced geometry.

“One of the coolest things that the Tools team at Pixar did for us was creating a system inside of Presto that would, by default, read in the entire set with the flattest version possible,” explains Moyer. Prior to that, for many years the layout artist or a camera artist would have to very carefully choose which objects they loaded in terms of how that would affect playability. By default now, sets come in completely flat and they don’t have any transforms on them. “You can then turn on some pieces if you want to focus on them. For example with a bookshelf, we could pop out an object, add transforms, and you can animate with it, but the rest of the set stays flat,” he added. This means the layout team can now do recordings with full fidelity and pass them on to editorial or animation. It avoids many issues that Moyer recalls of people saying, “Oh, did I forget to load the chair? Uh, what’s in the background? Am I getting a tangent in the background? So it’s really improved our workflow.”

The Antique Mall had between 10,000 and 11,000 props hand dressed into the set, based on about 2,000 uniquely modeled prop assets (instancing). The Second Chance Antiques covered the equivalent of about 8,000 square feet of retail space. The carnival had over 44,000 light bulbs and around 2,000 characters walking around the carnival and town.

It was only possible to render all of this due to Pixar’s adoption of a new noise reduction tool from Disney Studios Research in Zurich. They had previously used noise reduction software on films like Finding Dory but on Toy Story 4 they switched to the new research from the Zurich team. Moyer loved the quality of the results it output. It is possible to have noise reduction tools that loose detail but this was not the case with this new tool as it is very good at preserving detail. It halved render times and still produced brilliant fine detail, with Moyer joking that he, “kept describing it as ‘I’m eating a bucket of chicken that makes you lose weight!’ It was just magic and the imagery is incredible. It would turn a tiny light flickering patch of noise into this perfectly beautiful bokeh, almost all the bokeh that you see in the film are in render.”

The Zurich team wrote it using machine learning. The Pixar team would, “basically render out albedo, primary spec, secondary specular, primary diffuse, secondary diffuse, etc. and feed it to the denoiser which just basically does,… magic.”

Duke Caboom

Duke Caboom, (Keanu Reeves) – Canada’s greatest stuntman, and the antique store’s most beloved daredevil, joins Toy Story 4 as one of a series of new toys which cross paths with Woody and Buzz during their summer vacation.

Woody and Bo turn to a 1970s toy Duke Caboom for help. The toy is based on Canada’s greatest stuntman, (your personal experience may vary). Duke comes with a powerful stunt-cycle, that he’s always prepared to show off with a swagger, along with his stunt poses.

The motorcycle daredevil was partly procedurally textured to produce the consistent scale and detail of wear and aging. The character texturing artist who worked on Duke did so inside of Mari. “They actually spent a great deal of time instead of writing procedural shaders for Renderman, the way that our set’s department normally to,  they actually went in,  and wrote procedural shaders inside of Mari” explains Moyer. “They wrote global materials inside Mari so that they could do wear and woodgrain etc, and see it very quickly”. This allowed them to very quickly work up a believable fast wear and age pass for the character.  Duke’s cloth was also specially simulated, especially his cape. Given his size the threads scale make the cape heavy with large folds and “so his cape just sticks out his back,” comments Moyer.