How We Taught Neural Networks to Reproduce Physically Accurate Textures

When physics meets neural networks: how we taught neural networks to reproduce physically accurate textures

Computer graphics lives on tradeoffs. You want photorealism, so you build sophisticated lighting models. But every frame has a deadline — 16ms for 60fps — and the math has to fit inside that budget. Materials are where this tension hits hardest.

Film solves this by ignoring the clock. A single frame can render for hours. Artists build enormous material graphs, wiring together dozens of textures and math nodes to get the scuffs on an old couch or the reflections on a polished car just right. These graphs model real physics: light passing through paint layers, scattering in dust, bouncing off microscopic scratches. The output is indistinguishable from a photograph.

Games can't wait hours. Every frame needs to be done in 16 milliseconds or faster. So developers cut corners: pre-baked lighting (lightmaps), simpler material models, fewer layers. For a long time, the gap between offline and real-time rendering looked unbridgeable.

We wanted to try anyway. The idea: take a complex material built for film, train a neural network to replicate its behavior, and run that network fast enough for real-time use. It worked better than we expected.

Baking materials into neural networks

To understand what we did, it helps to know how high-quality materials are built.

In VFX and (to a lesser extent) game engines, the standard approach is layered materials. Think of an antique copper tray. An artist doesn't just paint a copper texture — they build a material that mimics the real physical structure:

• Base layer: the copper itself, with a specific color and sheen.
• Patina layer: greenish tarnish that builds up over time, thicker in recesses and thinner on raised areas.
• Varnish layer: a clear coating that gives strong but slightly blurred reflections.
• Dust layer: fine particles sitting on top.
• Scratch layer: cuts through everything above.

Each layer has its own mathematical model (BSDF — Bidirectional Scattering Distribution Function), controlled by masks, height maps, and procedural noise. The result is a graph with dozens of nodes and hundreds of connections. Evaluating this graph per pixel in real time is too expensive even for top-end GPUs.

Instead of evaluating this graph every frame, we train a neural network to approximate it. We call this baking:

1. The artist builds a reference material in their usual tool (Autodesk Maya with MaterialX, Houdini, etc.).
2. We generate training data by sampling the material repeatedly: random surface points (UV coordinates), random light directions, random camera directions.
3. For each sample, we evaluate the full material graph to get the ground-truth color and reflected light distribution.
4. The dataset (millions of samples) trains our neural model.
5. After training, the heavy material graph is discarded. Only the compact neural network and feature textures ship with the game.

This is a different kind of representation. Instead of simulating physical processes ("how light passes through layers"), we get a behavioral model ("how the surface looks under this lighting") that's hundreds of times smaller and an order of magnitude faster to evaluate.

How a neural material works, step by step

To be clear: this isn't one giant network for the whole scene. Each material gets its own compact neural model.

Rendering one pixel with a neural material goes like this:

Stage 1: Input data When a ray hits an object with a neural material, the shader gathers information about the hit point and sends it to the network:

• Texture coordinates (UV): where on the surface the ray landed. Used to look up the feature map.
• Directions: the incoming light direction ($\omega_{i}$) and the view direction ($\omega_{o}$).
• Additional attributes: normal maps for geometry correction (though the network can learn these effects on its own).

Stage 2: Feature map — the material's memory Feeding raw UV coordinates straight into a big network doesn't work well. The network would need enormous capacity to memorize every variation across the surface. Instead, we use a special texture called a feature map.

Unlike standard textures that store RGB colors or material parameters (roughness, metalness), our feature map stores latent vectors: compact multi-dimensional vectors (typically 8 values) that encode the material's properties at each point. An encoder, trained together with the decoder, learns to compress the full material complexity into these vectors.

Stage 3: Neural decoder — turning features into color The latent vector for a given point is just a list of numbers on its own. The Neural Decoder — a small fully connected network (multi-layer perceptron) — converts it into something useful. It takes the latent vector, combines it with the light and camera directions, and outputs the physical values the renderer needs. Getting this decoder's architecture right is what makes the whole system work.

Cooperative Vectors: running neural networks inside shaders

Designing the network was half the problem. The harder part was running it inside a GPU shader, hundreds of thousands of times per frame, within a real-time budget.

This is where Cooperative Vectors come in.

The problem

Neural networks normally run on GPU tensor cores through libraries like cuDNN and frameworks like TensorFlow or PyTorch. These are built for large batches and high throughput. Shaders are different: they're small programs running per-pixel or per-ray, and historically they've had no way to access tensor cores.

The traditional SIMT model needs a full warp (32 threads executing the same instruction) to use tensor cores efficiently. Ray tracing processes threads independently, so assembling full warps is nearly impossible. On top of that, tensor cores are tuned for matrix-matrix multiplication, but each ray thread only needs vector-matrix multiplication, which wastes compute resources.

The solution: Cooperative Vectors API

Cooperative Vectors is a new API being developed by AMD with Microsoft, integrated into DirectX. It's built for matrix-vector multiplication in graphics workloads, where each thread needs to multiply a vector by a relatively small matrix (like 128x128 or 64x64). Threads work together on these operations, which is where the "cooperative" name comes from.

Memory layout

Network weights and feature map data are packed into GPU memory buffers using layouts optimized for the access patterns Cooperative Vectors expects.

Divergence

Ray tracing is inherently divergent: one ray hits metal, the next hits fabric. Divergent execution hurts GPU performance. Cooperative Vectors handles divergence within a warp, though with some overhead.

To minimize this, we use Shader Execution Reordering (SER), which groups rays by material type. This gives Cooperative Vectors what it needs:

1. The same network weights across all threads in the warp.
2. A full warp of active threads.

Performance

Our neural shaders run more than 10x faster than equivalent non-neural layered materials implemented as traditional shader code. Fast enough to use cinematic-quality materials in games, which wasn't practical before.

Scaling to real scenes

A real scene has dozens or hundreds of unique materials. Loading a separate neural network for each one would blow out memory.

We use three strategies to keep things manageable:

1. Same architecture, different weights. All material networks share one architecture. Only the trained weights differ, which makes storage and switching efficient.
2. Shared direction interpretation. The part of the network that interprets light and view directions can be shared across materials. Only the feature maps are unique.
3. Scene-specific compilation. At runtime, we include only the materials actually used in the current scene or level and compile a shader for exactly that set.

The system works in practice, not just in benchmarks. You could build a game where a rusty nail and a velvet curtain both have physically correct surfaces that respond to lighting properly, and still hit a high frame rate.

3D Graphics Development

Need real-time neural materials or advanced 3D rendering? We build production-ready graphics solutions.

Explore the service

What's next

Neural materials connect offline and real-time rendering. Artists can build complex materials and ship them as neural models that run at interactive speeds. Engineers get better visual quality without tanking frame rates.

There's more to do: smaller models, materials that change over time (fading in sunlight, getting wet in rain). But the core idea works. Neural representations can produce physically accurate materials in real time, and they're practical enough to ship.