NVIDIA's Déjà View Beats Billion-Parameter 3D Models With Just 117M Parameters

NVIDIA Research has introduced Déjà View, a multi-view 3D reconstruction model that repeatedly applies one shared transformer block. The 117 million-parameter network matches or exceeds much larger feed-forward baselines in the reported benchmarks, while exposing its number of refinement steps as an inference-time control for latency and output quality.

The accompanying paper is listed as a NeurIPS 2026 oral, and NVIDIA has published the implementation in a GitHub repository. Given several unposed photographs of a scene, the model estimates dense per-view depth and camera rays used to recover camera poses and consistent 3D geometry.

One block, many refinements

Each input image first passes through a pretrained DINOv2 encoder, which converts pixels into visual feature tokens. Déjà View then runs a shared transformer block K times, updating the scene representation after every pass.

The block combines frame attention, which processes information within each image, with global attention that exchanges information across views. Every pass also receives a continuous-time value representing its position in the refinement schedule, allowing the shared weights to adjust their behavior as the reconstruction develops. Two lightweight heads decode the final state into dense depth and ray maps.

Conventional reconstruction transformers assign separate parameters to successive layers. Déjà View keeps the learned weight set fixed as K increases, so additional refinement adds computation and latency without enlarging the checkpoint.

Training samples a different K for each batch from a defined range. A single checkpoint can therefore run at supported step counts within that range. The project demonstration uses lower values such as K = 4 for faster results and higher values such as K = 16 for more refined depth and camera estimates.

Smaller weights, competitive results

Across five reported reconstruction benchmarks covering indoor, outdoor, object-centric, and driving scenes, Déjà View leads the compared models in average inlier ratio and pose AUC@30 while using the fewest parameters. Average inlier ratio measures the share of predictions within each benchmark’s correctness threshold. Pose AUC@30 summarizes camera-pose accuracy for errors up to 30 degrees.

The memory estimates cover raw model weights and exclude activations, framework overhead, inputs, and output tensors. Compared with pi-cubed and Depth Anything 3 (G), Déjà View has about one-eighth to one-tenth as many parameters and uses roughly 43% to 53% of their reported compute.

Shared recurrence earns its place

A repeatedly applied, weight-tied transformer resembles a recurrent neural network operating over visual tokens. NVIDIA’s ablation compares the shared loop with an otherwise similar design that assigns independent parameters to each step under matched training data and compute. The recurrent version performs better in the reported experiments.

The comparison suggests that repeated application of a learned refinement operator provides a useful inductive bias for multi-view geometry. It also narrows the explanation for the model’s performance: parameter sharing contributes more than checkpoint compression alone under the tested conditions.

Where the model fits

The 117 million-parameter checkpoint reduces weight memory for systems that cannot accommodate billion-parameter reconstruction models. Its adjustable step count also allows one deployment to serve several latency tiers, provided each selected value of K falls within the range used during training.

• Photogrammetry: generate initial geometry and camera estimates before optional structure-from-motion refinement.
• Robotics and augmented reality: recover depth and relative camera poses from images without known poses.
• NeRF and Gaussian splatting: initialize scene optimization with estimated depth and camera parameters.
• Mapping: process outdoor and driving imagery represented in the reported evaluation suite.

The costs move with K

Every refinement step depends on the preceding state, so latency grows with K. Higher step counts can reduce or erase some of the compute advantage over larger baselines even though the parameter count and raw weight memory remain fixed.

Runtime memory also depends on image resolution, token count, number of views, attention implementation, and retained activations. The parameter table therefore describes checkpoint storage rather than total GPU memory required for inference.

The benchmark averages establish performance on the five evaluated datasets but do not characterize every failure mode. Texture-poor surfaces, repeated patterns, motion between views, severe occlusion, unusual camera optics, and large viewpoint changes require separate testing on the intended data. Feed-forward predictions may also benefit from downstream per-scene optimization when an application prioritizes geometric precision over latency.

Checks before integration

Production evaluation should confirm the repository’s license, checkpoint terms, preprocessing requirements, supported image sizes and view counts, coordinate conventions, and depth-scale behavior. Teams should also benchmark several values of K on target hardware because FLOP counts do not capture memory transfers, kernel efficiency, or end-to-end latency.

The project page includes an interactive viewer for comparing refinement counts, while the paper and repository provide the architecture, reported experiments, and implementation needed to reproduce the results.