Game of Thrones' Battle of Winterfell was powered by artificial intelligence and horse treadmills

The eighth and final season of HBO's smash hit Game of Thrones was truncated at just six episodes, but producers D.B. Weiss and Dan Benioff made them count. Vast expanses of land that once took half a season to travel were now traversed in the course of several scenes, characters developed at preternatural paces, and the centerpiece battles were bigger than ever.

To help execute the massive clashes between Westeros' united armies and the zombified hordes from beyond the wall, the Thrones team turned to Weta, the digital effects powerhouse in New Zealand that has the market cornered on massive, monstrous brawls. Weta's aptly-named Massive simulation software, developed initially for the Lord of the Rings movies, is now applied across all manner of movies and TV shows, as it can create gigantic crowds of digital characters whose moves are governed by artificial intelligence. On Thrones, Weta was especially instrumental in creating the armies that clashed in the epic Battle of Winterfell.

In that series-shaking clash, the combined armies of Westeros took on the Night King, Whitewalkers, and zombified wights, who spilled into Winterfell by the thousands. Overall, Weta created an army of 30,000 Wights as well as the thousands of Dothraki that stood on the front lines against them. This was the most Weta had ever worked on Thrones, so the artists there had plenty of source material with which to work.

"For the wights, we were looking a lot at the Battle of Hardhome, where there's a lot of interesting sort of types of motion that the wights do," VFX supervisor Martin Hill told SYFY WIRE. "They all stand on top of cliffs, and then they stream off the cliff and crash against the ground. And then there's a beat where they kind of get up and then charge."

Hill gave SYFY WIRE an inside look at how Weta uses both math and artistic vision to give life to the undead.

You simulate hordes of wights at once — usually simulations are based on logic, but these are zombie-type creatures. So how do you decide how they'll move in these large groups?

They move in quite a sort of different way from your traditional zombie. They seem to have more control over their actions. They're heading definitely towards a target. They can move pretty fast, although some of them are more diminished, broken, can't move as fast. That gives them all manner of interesting limps and sort of loping gaits and strange sort of movements.

So, when they're stacking up against the walls of Winterfell, they're all hungry to get inside, but not to the extent where they'll pull another wight down to get in front of him. However, they will pull on a wight to climb up over him to get up faster, but they're not trying to get one-up over the wight next to them.

We've got a massive motion caption library here, but there were some things we needed to go back out and capture on our motion capture stage. And one of those things was all those towers, just getting that kind of loping, broken gait motion clambering over other wights up a 45-degree slope which then we could re-angle in the Massive software to whatever sort of angle we needed.

Credit: Weta

What are the parameters for logical behavior? Is it quantitative — you tell the computer that a character can only move within a perimeter — or is it more qualitative?

It's quite abstract. Massive runs under essentially a fuzzy logic, artificial intelligent algorithm. An agent is a fundamental unit. So, a Wight would be an agent. And generally what you give each wight is conceptually a field of view, so it can see in front of it, and it can hear all the way around it. So, it's reacting to its environment, depending on what it sees and kind of what it hears. That gives it the basis of going, well, you know, I can't run forward because I'm going to run into a brick wall. Or, you know, I have to run around these guys. Or, hey, I'm coming up to a wight who's up against a wall, I need to start climbing now.

Then, that all gets fed into the motion capture library and triggers different motion cycles or different parts of the capture library. So, a Wight knows, a sort of change from kind of a loping, broken run, and then trip on a bit of ground because the terrain changes a little bit, and then it knows it needs to start climbing. There are other triggers like, at this point it falls because someone on a filmed plate element hits them with a sword, so they fall down.

We use the ragdoll simulation, the physics simulation in Massive, as well, to break Wights apart, have them break apart into bones and actually have all their different bones snap. So then, it would just fall under gravity and bounce around on the ground. That was a really effective way of killing many, many Wights in the distance. The ones we've done in a much more hero ways were actually shot on the battlements, where a Wight getting shot with a flaming arrow. You can see the burst of fire that goes inside him and you can see his ribcage light up from the inside and it looks like a barbecue.

And is that gory stuff keyframe-animated or is it also simulated?

So 99.99% of them are in Massive and then there are hero shots [that are not simulated]. The characters pretty much have to be named before you get that. If you imagine the top of the pile of wights against Winterfell, the top two going over the wall, would either be close-up, either shot plate elements or CG [animated].

So how big is this motion capture library?

They're at the point of not really being countable. I would need to double check on that. What we generally do is motion capture stuff for upcoming projects sort of separate from the project, and then do more if the specific motion capture for the project [requires it], it really depends on the IP of the project.

Who performs all the actions for this giant library?

We have of a pool of talent mostly who are actors or stunt actors, stunt performers, or dancers, or gymnasts, or acrobats or you know, some ways physical performers and, you know, we hire them in as any other sort of acting gig and you know, put them through direction to create or enhance our motion.

How about the horses? Do you have go-to horse actors?

There's a veterinary university in Palmerston North that has a horse treadmill, which we use for motion capture. It's an enormous piece of equipment where horses can go from sort of walks to trots to canters to gallops. You stick on as best you can the motion tracking markers and set up the cameras in this sort of barn and then you motion capture the horses and hope you put on enough markers that when you get to full gallop and lots of them stay on that you get a good result.

I read that some of the horse-riding Dothraki were based on models you had for Lord of the Rings battles.

There's a definite sort of connection between say the Battle of Helms Deep in Rings and what we just did on the Long Night in episode 3 Game of Thrones.

So on the other end of things is the "ragdoll simulation," which is kind of the opposite of having control, but it's still simulated.

So the ragdoll simulation is essentially a broad name for physics-based simulation of soft bodies, generally people or animals, that aren't running under their own muscle power. You're using it when people fall off cliffs or die or collapse or you know, and they stop holding themselves up essentially. So, that's how the term ragdoll comes in. Essentially, it's like, you know, the analogy would be if you've got a puppet and you cut the strings, it turns into a ragdoll and just collapses.

One of the things we used it for in Thrones was the dragon's fire strafing through the armies, because all the Wights had just been thrown up into the air and at that point, you don't really need — or want — a logic to how they're going to move in that situation. We're not going to be motion capturing throwing people up into the air. So, they're just obeying the rules of physics. They're dipping and flying a parabola and their arms are going to flail depending on the masses of the different parts of their body and the limitations of each of their joints and their spine.

In broad strokes, it's very much a different technique to the crowd type simulations. However, the massive team have some really elegant ways of being able to blend between the two different bits of motion so it doesn't suddenly click from one method to another. They make sure it's seamless so you don't get a jolt in performance. The other thing the ragdoll does is it collides, so you can do things like you know, throw a trebuchet boulder into a crowd of Wights and they'll just be knocked like bowling pins around and that would all be sort of ragdoll simulation.

You also used simulations on dragons — how does that work, when there aren't exactly dragons to base it on?

So the dragons that we had, say for episode 4, we animated all the motion of Rhaegal and Drogon flying around Dragonstone and then Rhaegal's death. So to create that, that's all keyframe animation. On this show, I'll lead with Dave Clayton, who did a lot of the hero animation stuff.

What we essentially do is always try and base things on reality in terms of creatures as much as possible. So, obviously, there is no such thing as a dragon, but there are clearly analogs in the natural world. We work on it as if we would work on say an albatross or some kind of lizard and look at those sort of components. What is their skeletal structure like? Where are their muscle connections? Where are their tendon connections? And we build them from the ground up. You build a skeleton, essentially. You build the muscles and then you put a sort of skin-fashioned type layer in and then you put the skin and all the scales and things over the top.

The whole thing sits on top of the animation. So, you know, if Dave's animating a flap of the wings, he doesn't have to worry about getting all of that flapping down, you know, the muscles and the pecs and the tense and the traps for relax on the way down. That will happen in a kind of a procedural creature simulation mode which will then kind of deform. So that happens after animation. That's kind of secondary animation on top.