Shanghai Jiao Tong's Rein3D Turns One Photo Into a Walkable 3D Room
A restore-and-refine pipeline turns a single 360-degree photo into an explorable indoor scene by pairing 3D Gaussian Splatting with a panoramic video diffusion model.
ProTakeaways
- Rein3D reconstructs full 360-degree indoor scenes from a single RGB-D panorama using a restore-and-refine pipeline.
- A coarse 3DGS is rendered along radial trajectories to expose holes, then repaired by a panoramic video-to-video diffusion model.
- New PanoV2V-15K dataset provides 15,050 paired clean-degraded 360 videos across bedrooms, kitchens, and living rooms.
- Restored frames feed back as pseudo-ground-truths to refine a single global 3D Gaussian Splatting representation.
- Novel-view PSNR jumps from 15.05 to 17.64 on 3D-FRONT versus EmbodiedGen; FVD drops to 9.82.
- Backbone is Wan2.1-1.3B DiT with Context Anchor conditioning, latitude-aware noise sampling, and latitude-decay loss.
Turning a single panoramic photo of a room into a full 3D environment you can walk through has always been a losing battle against geometry. The moment the virtual camera steps away from where the picture was taken, huge regions the lens never saw appear as holes, warped surfaces, or floating debris. A new paper from Shanghai Jiao Tong University, Manycore Tech, D-Robotics, and the University of Hong Kong proposes Rein3D, a framework that reframes this problem as video restoration rather than novel-view hallucination.
The hole-filling problem nobody has cracked
Embodied AI and VR pipelines increasingly need high-quality 3D indoor scenes synthesized from sparse inputs, yet existing approaches struggle to infer the missing geometry in large unseen areas while keeping global consistency, often producing locally plausible but globally incoherent reconstructions. A panorama captures everything around one point, but translate the camera even a meter and walls behind furniture, floor patches under tables, and entire back-of-room regions become visible for the first time and need to be invented.
Prior work has tried two main tactics. Image-based diffusion methods generate one novel view at a time, but the resulting frames tend to disagree with each other when fused into a global 3D model. Video diffusion approaches produce more temporally coherent results, though standard video models operate on narrow field-of-view clips, so covering a full room requires stitching many trajectories together, which is expensive and still drifts geometrically.
The restore-and-refine trick
Rein3D formulates single-panorama 3D scene completion as 3D-prior-guided panoramic video restoration followed by global 3D Gaussian Splatting refinement. Instead of asking a model to dream up individual novel views, the pipeline deliberately produces a broken 3D scene first, then teaches a video model to fix it.