Skip to main content
Back

NVIDIA's WorldTrace Fixes AI Video Memory, Boosting Scene Recall by 32.6%

NVIDIA Research's WorldTrace adds training-free addressable memory to video world models, boosting scene recall 32.6% over sliding-window caches on LoopBench.

Paper, Full Access
  • NVIDIA Research's WorldTrace adds training-free addressable memory to video world models at NeurIPS 2026.
  • Scene recall improves 32.6% in DINOv2 similarity over sliding-window caches on LoopBench, same attended-cache size.
  • Summary slots use rank-based virtual positions, keeping RoPE offsets inside the trained range at any horizon.
  • Keys are averaged in canonical (unrotated) space to prevent phase cancellation between distant timestamps.
  • At 128 chunks on Matrix-Game-2, WorldTrace scores 0.698 DINOv2 versus 0.251 for sliding window.
  • Costs 3.1 GB extra GPU memory and about 27% wall-clock overhead; 360° pans remain hardest.

Interactive video world models often lose a place after the camera leaves it. When an agent returns, objects, textures, and room layouts may differ from the earlier view. NVIDIA Research has proposed WorldTrace, an inference-time memory framework that keeps compressed history addressable after a rollout exceeds the horizon used during training. The method changes cache management while leaving the model’s weights fixed.

NVIDIA reports a 32.6% improvement in average DINOv2 recall similarity over a sliding-window cache under the same six-frame attention budget. DINOv2 recall compares feature representations from the original and returned views of a location. Higher scores indicate a closer visual match.

Why old frames become unreachable

Autoregressive video generators use a key-value (KV) cache to carry generated context forward. Keys help attention find relevant tokens, while values contain the corresponding representations. Temporal Rotary Positional Embedding, or RoPE, encodes order by rotating query and key vectors according to their positions. Once generation passes the model’s training horizon, two mechanisms reduce retrieval quality:

  • Out-of-range positions: Older frames acquire RoPE offsets beyond those seen during training, making their keys harder for attention to retrieve reliably.
  • Phase-canceling compression: Averaging keys after RoPE combines vectors rotated by different angles. Their components can partially cancel, weakening the resulting summary.

Rank positions keep memory in range

WorldTrace combines a small recent window with compressed summary slots and selected verbatim landmarks. Each summary slot receives a virtual position based on its rank in the cache. As generation advances, the slot maintains a fixed offset from the current query and remains within the model’s trained RoPE range.

Before keys are merged, WorldTrace removes their positional rotation and maps them into a canonical space. It averages the canonical keys, then applies the rotation associated with the summary slot’s virtual position. This sequence avoids mixing incompatible positional phases.

WorldTrace cache architecture with summary slots, landmarks, and a recent window
WorldTrace preserves recent frames, compressed field summaries, and selected scene landmarks.

WorldTrace-Field merges redundant history into summary slots. WorldTrace-Landmark preserves scene-entry frames whose canonical keys cross a cosine-distance threshold relative to the preceding frame, signaling a substantial change in visual content. Candidate merges and evictions are scored by their attention-response error on sampled queries, which estimates how much each operation would alter the model’s attention output. Key-value similarity then determines how selected entries are combined.

LoopBench forces a return trip

The research team built LoopBench to test whether a model can revisit its own generated scenes. A rollout passes through waypoints, returns to an earlier location, and compares the new view with the original using DINOv2 similarity and Return LPIPS. LPIPS measures perceptual distance, so lower values indicate a closer match.

  • Topology: ABA, ABCA, and ABCDA routes add progressively more intermediate waypoints.
  • Rollout length: ABA loops extend from 8 to 16 to 32 generated chunks.
  • Orientation: The camera rotates 90°, 180°, or 360° at one location.
  • Multiple revisits: Routes including ABABA and ABCBA return to generated scenes more than once.
LoopBench examples covering route topology, rollout length, camera rotation, and repeated visits
LoopBench varies route complexity, duration, camera movement, and revisit frequency.

On the 1.3-billion-parameter Matrix-Game-2 model, every method was restricted to the same six-frame attended window. WorldTrace produced the highest mean DINOv2 score among the compared methods in all 12 benchmark conditions, raising the tier-averaged score from 0.587 to 0.778.

LoopBench DINOv2 ablation results
Cache configuration Mean DINOv2
Sliding window 0.587
Field summaries only 0.576
Single verbatim landmark 0.715
WorldTrace combined 0.778

The single-landmark configuration accounts for most of the improvement, while field summaries alone score slightly below the sliding-window baseline. Combining landmarks with field summaries adds another 0.063 to the mean DINOv2 score.

Longer rollouts widen the gap. At 128 chunks, WorldTrace records 0.698 DINOv2 and 0.572 LPIPS, compared with 0.251 DINOv2 and 0.751 LPIPS for the sliding-window cache.

The gains narrow in hard turns

Return LPIPS improves over the sliding window in every LoopBench condition except the 180° rotation, where WorldTrace scores 0.649 against 0.634. The 360° turn remains the hardest condition: WorldTrace leads only marginally on DINOv2, with the difference falling within one standard error.

The resource comparison also requires a distinction between attention budget and total memory use. All methods attend to six frames, but WorldTrace’s verbatim bank consumes an additional fixed 3.1 GB of GPU memory. Scoring that bank adds roughly 27% to generator wall-clock time, according to the paper’s appendix.

MemRoPE, another training-free positional baseline, remains within 0.046 DINOv2 of the sliding window. The reported evaluation also covers one 1.3-billion-parameter generator, leaving performance across other architectures and model scales unmeasured.

The integration lives in the cache

Long-horizon consistency constrains interactive generators used for games, robotics simulation, and agent training. Returning to a location requires stable geometry and appearance across many generation steps. WorldTrace addresses that requirement by managing the KV cache as an addressable memory containing recent context, compressed history, and episodic landmarks.

For developers using an autoregressive video model with temporal RoPE, the high-level integration has four parts:

  1. Reserve attended slots for recent frames, canonical-space summaries, and selected landmarks.
  2. Remove RoPE rotations before merging keys, then rotate each summary at its virtual cache position.
  3. Assign summary positions by cache rank to keep relative offsets within the trained range.
  4. Use attention-response error and key-value similarity to select merges, evictions, and verbatim landmarks.

The project page and NeurIPS 2026 poster provide implementation diagrams, benchmark details, and video comparisons of returned scenes.