Skip to main content
Back

Nanjing University's UniSplat Rebuilds 3D Scenes From Photos in 40 Milliseconds

UniSplat unifies geometry, appearance, and semantics from raw multi-view images without camera poses, setting new marks on 3D vision and robotics benchmarks.

Pro

Paper, Full Access
  • UniSplat is a feed-forward transformer that reconstructs 3D scenes from unposed images in 0.041 seconds, no SfM required.
  • A dual-masking strategy hides geometry-rich patches twice, forcing the decoder to reason about 3D structure from sparse cues.
  • Coarse-to-fine hierarchical Gaussians (anchor, semantic, appearance) resolve the granularity mismatch between smooth semantics and fine textures.
  • Pose-conditioned recalibration reprojects 3D point maps to 2D and matches renders, enforcing consistency across all decoder heads.
  • Hits 25.65 PSNR and 0.5625 mIoU on ScanNet, beats NoPoSplat, DUSt3R, MASt3R on pose estimation, project page at bobochow.github.io/UniSplat.
  • Frozen ViT-L encoder outperforms MAE, DINOv2, CLIP, VC-1, and SPA across 268 embodied AI tasks in VC-1, RLBench, LIBERO, and Meta-World.

Reconstructing a 3D scene from a handful of photos usually starts with a painful preamble: running Structure-from-Motion to figure out where each camera was, feeding those poses into a Gaussian Splatting or NeRF pipeline, and hoping the geometry, colors, and any semantic labels line up when you are done. A new CVPR 2026 paper from Nanjing University of Science and Technology and Zhejiang University proposes UniSplat, a single feed-forward transformer that skips the SfM step entirely and predicts geometry, appearance, and language-aligned semantics from unposed images in about 40 milliseconds.

Speed is only half the story. Prior pose-free methods like RayZer couple camera and scene recovery in a predict-then-render loop, while SelfSplat predicts depth and pose with separate modules, yielding interpretable but less coherent alignment. UniSplat argues that the real bottleneck in this line of work is inconsistency between what each head predicts, and it introduces three mechanisms to force geometry, color, and semantics to agree on the same 3D scene.

Why unposed multi-view is still hard

Feed-forward 3D reconstruction has been converging on a common recipe: predict per-pixel Gaussians in a canonical coordinate frame and rasterize them for novel-view synthesis. NoPoSplat, for example, anchors one input view's local camera coordinates as the canonical space and predicts Gaussian primitives for all views within it, avoiding errors from per-frame Gaussians and pose estimation. That works well for rendering, but the resulting features are often shallow when you ask them to also carry semantic meaning or serve as a backbone for a robot policy.

The authors identify three specific failure modes in self-supervised pose-free pipelines: weak geometry induction (the encoder learns view-aligned rather than spatially grounded features), limited appearance detail, and mismatches between where geometry says something is and where semantics says something is. UniSplat targets each one with a dedicated component.