Real-Time Character Animation with Machine Learning - An Overview of PFNN, MANN, and LMM

We implemented Learned Motion Matching, developed a plugin for Unreal Engine 5, and integrated trained models into a real-time animation pipeline. Next, we will review existing character animation solutions based on neural networks.

Introduction

Until recently, the standard approaches in character animation included keyframe animation or procedural animation — a family of fundamentally different methods based on inverse kinematics, ragdoll physics, or more complex systems (e.g., GTA IV — Euphoria). However, despite their widespread use, these methods have significant drawbacks: unrealistic results, high cost, limited expressiveness, reliance on manual labor, and difficulty maintaining a consistent artistic style. Then came motion matching, which delivered an entirely different level of animation quality, but only AAA-scale studios could afford such systems. Moreover, motion matching is extremely demanding on memory due to the need to store the entire animation library in RAM.

Some of these shortcomings can be naturally addressed by applying machine learning, thanks to its low memory consumption, scalability with respect to data, and ability to generalize. Today, a new shift can be observed: more and more tasks related to character movement, facial expressions, and behavior are being handed over to machine learning models. The reason is simple — games, VR/AR systems, virtual actors, and interactive simulations require not just beautiful animation, but realistic real-time behavior that adapts to the environment and user actions, which traditional systems cannot provide.

Neural networks can learn from large motion capture datasets, predict movement for upcoming frames, synthesize transitions between poses, control gait, balance, obstacle reactions, and even facial expressions synchronized with speech. The result is animation that looks natural while being generated on the fly — without pre-recorded clips. However, new capabilities come with new challenges: performance, stability, control over results, data quality requirements, and integration into existing pipelines.

In this article, I will present some of the existing machine learning-based solutions for character animation. Since the approaches discussed below are closely related to motion matching technology, I will begin with a brief overview of it.

Motion matching is a search algorithm that selects the pose from an animation database that best fits the current context. At present, this technology delivers the most responsive and plausible animations. It has been used in the following projects: The Last of Us, For Honor, Fortnite.

Instead of describing animation through a state graph, motion matching allows animators to specify pose parameters (features), and the best-matching pose is automatically selected using a nearest-neighbor search. This means that animation quality and variety depend directly on the size of the database. Since the algorithm takes poses from the database as-is, applying only blending and post-processing, the quality of the original animation is preserved, and animators retain control over the result. The motion matching process can be summarized as follows:

1. Compute the desired movement trajectory
2. Search the database for the most suitable pose, considering pose parameters and the movement trajectory
3. Assign the closest matching pose to the character
4. Post-processing
5. Repeat steps 1–4

The process of adding new animation material to the database is straightforward and can be performed and debugged in real time. As a result, memory consumption grows linearly with the amount of data and the number of pose search parameters. Ultimately, the task comes down to finding a balance between result quality, memory consumption, and computational performance.

Thus, the following key disadvantages of motion matching can be identified:

1. The need for a massive dataset — for example, in TLOU 2, Ellie and Abby alone have a total of 2,627 animation clips — 2 hours and 17 minutes of animations
2. Lack of generativity

Typical Pipeline

The pipeline is conceptually the same for all three architectures and looks as follows:

1. Preprocessing: at this stage, training data is prepared, controller parameters are automatically extracted, and terrain is matched to movement using a separate heightmap
2. Training: the neural network learns to output character movement at each frame using controller parameters
3. Real-time inference: input data for the neural network (controller and environment) is collected and fed into the system, which determines the character's movement

The source training data is standard motion capture, recorded as a long sequence describing various gaits and movement directions. During preprocessing, the entire sequence is mirrored to augment the dataset.

Bone position and velocity data is used in an autoregressive manner — the computed result from the previous frame is used as input for the next one.

Phase-functioned neural networks for character control (PFNN)

PFNN is a neural network architecture that operates by generating the weights of a regression network at each animation frame as a function of phase — a variable representing the movement cycle time. These weights are then used to perform regression from controller parameters at the current frame to the corresponding character pose. The proposed architecture is suitable for real-time operation thanks to its computational speed and lightweight memory footprint. Additionally, by sacrificing compactness, performance can be further improved through precomputation of the phase function.

A significant improvement in animation expressiveness is achieved through the dynamic adjustment of neural network weights based on the phase function — this approach allows the network to learn from a high-dimensional dataset where environmental geometry and human motion data are interconnected. The PFNN architecture avoids mixing data from different phases, and the regression function changes smoothly over time depending on the phase. After training, the system can automatically generate appropriate and expressive movements for a character traversing rough terrain, jumping, and avoiding obstacles — in both natural and urban environments.

The input to the neural network consists of the previous pose and controller input, while the output includes the phase change, the current character pose, and some additional parameters.

Along with the animation, controller parameters are provided as training data, consisting of the movement phase, gait semantic labels, character movement trajectory, and a heightmap along the trajectory.

• Phase labeling: performed semi-automatically. Foot contact with the surface is computed automatically via velocity and then manually corrected. Once contact data is obtained, phases are assigned: when the right foot contacts the surface, phase = 0, then the left foot = $\pi$, right foot again = $2\pi$ ($0 \leq p \leq 2\pi$). Interpolation is performed between frames.
• Gait labeling: performed manually, represented as a binary vector to disambiguate similar movement types and describe specific movement scenarios.
• Trajectory and heightmap: the root transformation of the character describing the movement trajectory is extracted. Then, along the entire trajectory and perpendicular to it on both sides, the surface height is computed.

To describe the character's state, local (relative to root transformation) bone positions and velocities at the current animation frame are used. For trajectory construction, 5 frames from the future and 6 from the past are sampled, covering a total of 1 second of past movement and 0.9 seconds of future movement. For each sample, the position and direction of the trajectory relative to the root transformation are extracted, along with a binary vector describing the gait and the surface height under and beside the trajectory at a distance of 25 cm. Thus, the full input vector for the neural network for a single animation frame is:

• $i$ — current frame
• $i-1$ — previous frame
• $s$ — number of sampled frames (12)
• $b$ — number of bones in the skeleton
• $t_i^p \in R^{2s}$ — trajectory positions in the 2D horizontal plane
• $t_i^d \in R^{2s}$ — trajectory directions in the 2D horizontal plane
• $t_i^h \in R^{3s}$ — heights at points to the left, right, and center of the trajectory
• $t_i^g \in R^{5s}$ — semantic labels describing the character's gait and other information
• $j_{i-1}^p \in R^{3b}$ — local bone positions
• $j_{i-1}^v \in R^{3b}$ — local bone velocities

The original paper used the following labels for $t_i^g$:

1. idle
2. walking
3. running
4. jumping
5. crouching

The output of the neural network inference is the vector:

• $s$ — number of sampled frames (12)
• $b$ — number of bones in the skeleton
• $t_{i+1}^p \in R^{2s}$ — predicted trajectory positions at the next frame
• $t_{i+1}^d \in R^{2s}$ — predicted trajectory directions at the next frame
• $j_i^p \in R^{3b}$ — local bone positions
• $j_i^v \in R^{3b}$ — local bone velocities
• $j_i^a \in R^{3b}$ — bone angles expressed in exponential map form
• $\dot{r_i^x} \in R$ — linear velocity of the root transformation along the X axis relative to the "forward" direction
• $\dot{r_i^z} \in R$ — linear velocity of the root transformation along the Z axis relative to the "forward" direction
• $\dot{r_i^a} \in R$ — angular velocity of the root transformation around the vertical axis
• $\dot{p_i} \in R$ — phase change
• $c_i \in R^4$ — foot contact information (toe and heel) with the surface

The neural network consists of 3 layers:

The number of neurons in each layer is 512. The network weights $a$ are computed depending on parameters $\beta$ at each frame by a separate phase function $a=\Theta(p;\beta)$. This function can be another neural network or a Gaussian process, but the paper proposes using a cyclic cubic Catmull-Rom spline with 4 control points. This approach means that each control point $a_k$ represents a specific configuration of network weights $a$, and the function $\Theta$ performs smooth interpolation between these configurations.

$\Theta(p; \beta) = a_{k_1}$

$+w(\frac{1}{2}a_{k_2} - \frac{1}{2}a_{k_0})$

$+w^2(a_{k_0} - \frac{5}{2}a_{k_1} + 2a_{k_2} - \frac{1}{2}a_{k_3})$

$+w^3(\frac{3}{2}a_{k_1} - \frac{3}{2}a_{k_2} + \frac{1}{2}a_{k_3} - \frac{1}{2}a_{k_0})$

$w = \frac{4p}{2\pi} (mod 1)$

$k_n = \left[\frac{4p}{2\pi} \right] + n - 1 (mod 4)$

Training the neural network amounts to solving the optimization problem for the phase function parameters $\beta = \left\{ a_0 a_1 a_2 a_3 \right\}$. The following loss function is used:

• $\lVert Y - \Phi \left(X; \Theta\left(P; \beta\right)\right) \rVert_2^2$ — mean squared error
• $\gamma \lvert \beta \rvert_1$ — regularization introduced to prevent the weights from becoming too large ($\gamma = 0.01$)

Adam was chosen as the optimizer.

In real time, at each animation frame, the phase $p$ and vector $x$ are fed as input to the neural network. The velocity and direction of movement received from the controller for each future frame are blended with the data predicted by the neural network at the previous frame $(t_{i+1}^p t_{i+1}^d)$ using the following formula:

• $0 \leq t \leq 1$
• $\tau$ — additional bias controlling the character's responsiveness

Mode-adaptive neural networks for quadruped motion control (MANN)

This work extends the PFNN architecture to the animation of quadruped characters. Due to fundamental differences in movement patterns, it is impossible to define a single phase for all four limbs when the gait changes. Manual labeling of an unstructured dataset also becomes impractical. Therefore, the phase function is replaced by an additional neural network. The resulting system includes a motion prediction network and an expert selection network (gating network — similar to the mixture of experts architecture): while the first network predicts the character's state based on the previous frame's state and controller input, the second network dynamically updates the weights of the first network by selecting and mixing a new entity called "expert weights," each of which corresponds to a specific movement pattern. This architecture allows the networks to learn from data without labeled gaits, completely eliminating the phase labeling stage.

Movement classification is performed manually to identify 6 movement classes: locomotion, sitting, standing, waiting, lying down, and jumping. This is done so that in real time the user can specify the movement class via the controller.

The paper examines four gait types: walk, pace, trot, and gallop. Although the system does not require these labels for real-time character control, an analysis of the distribution of these types in the dataset is performed based on velocity calculations.

Input and output data are generally similar to PFNN: 5 future frames and 6 past frames are also sampled, the character's root transformation and movement direction are computed.

The input data vector:

• $i$ — current frame
• $i-1$ — previous frame
• $s$ — number of sampled frames (12)
• $b$ — number of bones
• $t_i^p \in R^{2s}$ — trajectory positions in the 2D horizontal plane
• $t_i^d \in R^{2s}$ — trajectory directions in the 2D horizontal plane
• $t_i^v \in R^{2s}$ — velocities at trajectory points in the 2D horizontal plane
• $t_i^{\hat v} \in R^{1s}$ — desired velocity at trajectory points
• $t_i^a \in R^{6s}$ — one-hot vector of movement classes at trajectory points
• $j_{i-1}^p \in R^{3b}$ — local bone positions
• $j_{i-1}^r \in R^{6b}$ — local bone rotations
• $j_{i-1}^v \in R^{3b}$ — local bone velocities

Adding bone rotations to the input vector allowed for more responsive animation.

The output of the neural network inference is the vector:

• $i$ — current frame
• $i+1$ — next frame
• $s$ — number of sampled frames (12)
• $b$ — number of bones
• $t_{i+1}^p \in R^{2s}$ — trajectory positions
• $t_{i+1}^d \in R^{2s}$ — trajectory directions
• $t_{i+1}^v \in R^{2s}$ — velocities at trajectory points
• $j_i^p \in R^{3b}$ — local bone positions
• $j_i^r \in R^{6b}$ — local bone rotations
• $j_i^v \in R^{3b}$ — local bone velocities
• $\dot r_i^x \in R$ — linear velocity of the root transformation along the X axis
• $\dot r_i^z \in R$ — linear velocity of the root transformation along the Z axis
• $\dot r_i^a ∈ R$ — angular velocity of the root transformation in the 2D horizontal plane

Bone rotations are represented as relative "up" and "forward" vectors to avoid quaternion interpolation issues during neural network training.

The motion prediction network architecture is similar to PFNN, but the weights $a$ are computed by mixing $K$ expert weights $\beta = \left\{ a_1, \dots, a_k \right\}$,

where $a = \sum_{i=1}^K \omega_i a_i$. $K$ is a tunable meta-parameter depending on the complexity and size of the training data. $\omega = \left\{ \omega_1, \dots , \omega_K \right\}$ are the mixing coefficients computed by the gating network.

The gating network architecture consists of 3 layers:

• $\hat x \in R^{19}$ — a subset of $x$ including foot bone velocities, current movement classes, and the desired character velocity.
• $\sigma(\cdot)$ — softmax function for normalizing inputs so that their sum equals 1, which is necessary for subsequent linear mixing

The network parameters $\mu$ are defined as follows:

• $h^{'}$ — number of neurons in hidden layers (32)

Training the network amounts to finding $Y$ for the corresponding $X$, which is a standard regression task. The following loss function is used (MSE between prediction and ground truth):

AdamWR was chosen as the optimizer.

Learned motion matching (LMM)

LMM is based on Ubisoft's own implementation of motion matching. Classic motion matching consists of 3 stages: Projection, Stepping, Decompression — for each of which a trained neural network is proposed as an alternative.

For the locomotion scenario, a 27-element vector per animation frame is proposed as pose parameters:

• $t^t \in R^6$ — trajectory positions in 2D projected onto the surface at 20, 40, 60 frames in the future (at 60 FPS)
• $t^d \in R^6$ — trajectory direction at 20, 40, 60 frames in the future
• $f^t \in R^6$ — local positions of foot bones
• $\dot{f^t} \in R^6$ — linear velocities of foot bones
• $\dot{h^t} \in R^3$ — linear velocity of the hip bone

Next, a vector containing all pose information for each animation frame is defined:

• $y^t \in R^3$ — local bone positions
• $y^r \in R^4$ — local bone rotations in axis-angle representation
• $\dot{y^t} \in R^3$ — linear bone velocities
• $\dot{y^r} \in R^3$ — angular bone velocities
• $\dot{r^t} \in R^3$ — linear velocities of root transformation
• $\dot{r^r} \in R^3$ — angular velocities of root transformation
• $o^*$ — task-specific additional data (e.g., foot contact information)

These vectors are computed for each frame, combined into matrices $X = \left[ x_0, x_1, \dots, x_{n-1} \right], Y = \left[ y_0, y_1, \dots, y_{n-1} \right]$, called the matching database and animation database respectively, and used in the training algorithm. In real time, every N frames or when the controller input changes significantly, a query vector $\hat x$ is constructed — analogous to the pose parameter vector — and fed as input to the Projection stage. Once a new frame is found, animation playback starts from it and a transition is inserted.

• Projection: nearest-neighbor search to find the pose parameter vector in $X$ that best matches $\hat x$.
• Stepping: advances the index in the matching database.
• Decompression: retrieves the pose from the animation database corresponding to the current index in the matching database.

By replacing each of these stages with a neural network, the need to store the matching database and animation database in memory is eliminated. To this end, 4 neural networks are introduced:

• Decompressor: eliminates the need to store $Y$ in memory, takes $x$ and a latent vector $z$ as input
• Compressor: acts as an encoder for finding $z$ by compressing $y$ into a lower-dimensional vector
• Stepper: together with the projector, eliminates the need to store $X$ in memory; learns the dynamics of the system by computing changes in $x_i$ and $z_i$ values to obtain $x_{i+1}$ and $z_{i+1}$ for the next frame
• Projector: emulates the nearest-neighbor search, takes $\hat x$ as input and predicts the closest $x_{k^*}$ and $z_{k^*}$

By combining these 4 neural networks, the result is learned motion matching, shown in the figure below.

Since the pose parameter vector typically does not contain enough information to derive the corresponding pose, a latent space $Z$ is introduced. Its values are obtained through the Compressor network — a mapping from $y_i$ to the corresponding $z_i$. This vector is then concatenated with $x_i$ and fed into the Decompressor, which attempts to reconstruct the original pose $y_i$. In this way, the network discovers what information is missing from the pose parameter vector $x$ and encodes it in $z$.

A key aspect of Decompressor training is the loss function designed to minimize the visual perception of error, which uses forward kinematics to evaluate the error in character space. Velocity-aware loss terms are also added so that the result changes smoothly over time.

After training, the Compressor network is not required for real-time operation, since it is only needed to compute $Z$ used for training the other networks.

The Stepper network is trained to take vectors $x_i$ and $z_i$ of the current frame as input and output a delta added to them to obtain vectors $x_{i+1}$ and $z_{i+1}$ for the next frame. A small window of $X$ and $Z$ vectors is taken, and the next values of pose parameters and latent variables are predicted and fed to the next frame.

Finally, the Projector network completely eliminates the need to store $X$ and $Z$ in memory. For its training, a vector $x$ is taken, Gaussian noise $n$ is sampled and scaled by random noise $n^\sigma$, the resulting value is added to $x$ to obtain $\hat x$, and the nearest $k^*$ is found via nearest-neighbor search. The Projector is then trained to output the corresponding pose parameter vectors $x_{k^*}$ and latent variables $z_{k^*}$.

By sampling noise of varying magnitude, the Projector is made robust to perturbations at different scales.

For all loss functions in all training algorithms, the coefficients $w*$ are chosen to equalize the initial values (at the 1st training iteration).

RAdam was chosen as the optimizer.

The number of layers, neurons, and activation functions for the networks are shown in the figure below:

The runtime operation of the architecture proceeds as follows:

1. $\hat x$ is formed and fed as input to the Projector, which outputs $x_{k^*}$ and $z_{k^*}$
2. The found $x_{k^*}$ and $z_{k^*}$ are fed as input to the Stepper, which advances them in time
3. The result from step 2 is fed as input to the Decompressor, which outputs the final character pose

Platform limitations prevent embedding a demonstration here; video materials can be found via the original sources.

Conclusion

This article did not include performance and quality evaluations of the discussed approaches — you can find those in the original papers in more detail.

Additionally, some important components (trajectory construction, character-to-heightmap matching) were not covered as they fall outside the scope of this article. In the next article, I plan to cover some of the omitted details.

References:

• Motion matching: Motion Matching and The Road to Next-Gen Animation
• The Last of us Part II: Bringing Allies to Life in the 'Last of Us Part II'
• PFNN: Phase-Functioned Neural Networks for Character Control
• MANN: Mode-Adaptive Neural Networks for Quadruped Motion Control
• LMM: Learned Motion Matching