Google DeepMind's D4RT Reconstructs Dynamic 4D Scenes 300x Faster

Google DeepMind has introduced D4RT, short for Dynamic 4D Reconstruction and Tracking, a single transformer that pulls geometry, motion, and camera parameters out of an ordinary video in one shot. In testing, it performed 18x to 300x faster than the previous state of the art. The work is being presented at CVPR 2026 and the technical report is on arXiv.

The pitch is simple: instead of stitching together a depth estimator, a point tracker, and a pose solver, you get one feedforward model that answers a single, very general question about any pixel at any time from any viewpoint. That reframing is what unlocks both the speed and the accuracy gains.

One question to rule four dimensions

D4RT operates as a unified encoder-decoder Transformer architecture. The encoder first processes the input video into a compressed representation of the scene's geometry and motion. Unlike older systems that employed separate modules for different tasks, D4RT calculates only what it needs using a flexible querying mechanism centered around a single, fundamental question: "Where is a given pixel from the video located in 3D space at an arbitrary time, as viewed from a chosen camera?"

That phrasing matters because it collapses three traditionally separate computer vision tasks into the same interface. You parameterize a query by a source pixel (u, v), a source timestep t_src, a target timestep t_tgt, and a target camera t_cam, and the decoder returns the 3D position. The query also carries a local image patch around the pixel for extra spatial context.

The key engineering insight is that queries are independent of each other. Because queries are independent, they can be processed in parallel on modern AI hardware. This makes D4RT extremely fast and scalable, whether it's tracking just a few points or reconstructing an entire scene. That sidesteps the dense per-frame decoding that bogs down most prior 4D systems, which have to commit to outputting a full depth map or flow field whether you need every pixel or not.

What it can actually do

By varying the query, the same weights handle a surprisingly wide menu of tasks:

• 3D point tracking: By querying a pixel's location across different time steps, D4RT can predict its 3D trajectory. Crucially, the target point does not need to be visible in the queried frame, so occlusions are handled implicitly.
• Point cloud reconstruction: By freezing time and the camera viewpoint, D4RT can directly generate the complete 3D structure of a scene, eliminating extra steps such as separate camera estimation or per-video iterative optimization.
• Camera pose estimation: By generating and aligning 3D snapshots of a single moment from different viewpoints, D4RT can easily recover the camera's trajectory.
• All-pixels tracking: the model produces a holistic scene reconstruction by predicting 3D tracks for every pixel in the video in world coordinates, with dynamic objects deduplicated rather than smeared across frames.

The speed numbers

The efficiency gains are what make this more than a benchmark paper. D4RT processed a one-minute video in roughly five seconds on a single TPU chip. Previous state-of-the-art methods could take up to ten minutes for the same task , an improvement of 120x. That moves dynamic scene reconstruction from an offline, batch-processing problem into something close to real-time territory.

Accuracy held up across standard benchmarks. In evaluations on the MPI Sintel benchmark featuring complex synthetic scenes with fast motion blur and non-rigid deformation, D4RT demonstrates superior fidelity compared to recent strong baselines. Using smart-glasses footage from the Aria Digital Twin dataset, D4RT achieves top-tier performance in 3D point tracking. On the RE10k dataset for camera pose, it also tops the leaderboard without test-time optimization.

Why dynamic scenes are the hard part

In order for it to understand a dynamic scene captured on a 2D video, an AI model must track every pixel of every object as it moves through the three dimensions of space and the fourth dimension of time. In addition, it must disentangle this motion from the motion of the camera, maintaining a coherent representation even when objects move behind one another or leave the frame entirely. Most prior pipelines either bolt depth, flow, and SLAM together (lots of glue code, lots of error accumulation) or run per-video iterative optimization (slow). D4RT does neither.

The qualitative wins matter too. Qualitative comparisons show that while other methods struggle with dynamic objects , often duplicating them or failing to reconstruct them entirely , D4RT maintains a solid, continuous understanding of the moving world. That duplication problem is a recurring failure mode in feedforward 4D systems, where a moving person ends up as a ghosted trail in the point cloud.

Where it slots into the stack

The architecture builds on the team's earlier Scene Representation Transformer work. The encoder produces what they call a Global Scene Representation, and the decoder is deliberately lightweight so that querying scales sub-linearly with output density. For users who only want a sparse track of a few key points, you pay almost nothing; for dense reconstruction, you parallelize queries across the chip.

The intended downstream targets are concrete:

• Robotics: spatial awareness for navigation and manipulation in environments with moving people and objects.
• Augmented reality: D4RT's efficiency contributes to making on-device deployment a tangible reality.
• World models: By effectively disentangling camera motion, object motion, and static geometry, D4RT brings us a step closer to AI that possesses a true "world model" of physical reality , a necessary step on the path to AGI.

What changes for the field

The assumption that multi-task 4D systems require multi-task decoders takes a hit here. D4RT shows that a single query interface, when parameterized over space, time, and viewpoint, subsumes depth, tracking, and pose without the usual accuracy tax. For anyone currently maintaining a frankenstack of MonST3R, CoTracker, and a SLAM module, the path forward likely involves rethinking that pipeline as a single queryable scene representation.

Code and weights are not announced in the materials available, and the work is currently a research release with the paper, project page, and a Google DeepMind blog post as the primary artifacts. If you work on robotics perception, AR scene understanding, or video-based world models, this is the paper to read closely while waiting to see whether a public checkpoint follows.