Skip to main content
Back

OneWorld Ditches 2D Tricks to Generate 3D Scenes With 40% Sharper Views

OneWorld runs diffusion inside a 3D feature space instead of 2D latents, cutting cross-view drift and lifting PSNR by over a point on RealEstate10K.

Pro

  • OneWorld runs diffusion inside a unified 3D feature space instead of 2D image/video latents for scene generation.
  • Core is 3D-URAE, built on the π³ foundation model with an appearance injection branch and DINOv2 semantic distillation.
  • Cross-View-Correspondence loss and Manifold-Drift Forcing fix cross-view drift and train-inference exposure bias.
  • Beats DepthSplat on RealEstate10K: PSNR 28.19 vs 27.02, LPIPS 0.102 vs 0.169+ under 8-view reconstruction.
  • NVS-Refined dataset already on Hugging Face, model weights promised soon.
  • Code repository at github.com/SensenGao/OneWorld, paper at arXiv 2603.16099.

Most diffusion-based 3D scene generators are secretly 2D models. They denoise in the latent space of an image or video VAE and hope the frames line up well enough to reconstruct a scene. A new paper called OneWorld ditches that indirection and runs diffusion directly inside a 3D feature space, then decodes the result into Gaussian splats. The authors report cleaner novel views, better cross-view consistency, and a solid jump over specialized reconstruction baselines like DepthSplat.

Why 2D latents keep breaking 3D scenes

A video VAE has no notion of a shared underlying geometry. When you denoise two frames of the same room, nothing in the loss forces the wall on the left to be the same wall on the right, and small per-view drifts compound as the camera moves.

Recent work has tried to patch this in various ways. FlashWorld diffuses in compressed video latents. Gen3R compresses 3D features to align a 3D foundation encoder with video latents, but can only generate geometry and appearance separately rather than jointly. OneWorld's pitch is simpler: generate directly in a unified 3D representation space, without compression or a split objective.

Teaser figure comparing OneWorld to FlashWorld and Gen3R with benchmark results

A 3D latent you can actually render

The core contribution is the 3D Unified Representation Autoencoder, or 3D-URAE. It starts from a pretrained 3D foundation model, specifically π³ (denoted F), which infers key 3D quantities of a scene from observed views together with their camera parameters. Foundation models like this are geometry-obsessed and tend to throw away texture and material detail, so the authors bolt two branches onto it: