NVIDIA's Lyra 2.0 Turns a Single Photo Into a Walkable 3D World
NVIDIA's new Lyra 2.0 turns a single photo into an explorable 3D world, exportable to Isaac Sim and real-time renderers.
Takeaways
- NVIDIA released Lyra 2.0, generating explorable 3D worlds from a single input image.
- Outputs 3D Gaussian Splats and meshes exportable to Isaac Sim for robotics simulation.
- Built on WAN-14B backbone, trained on 32 H100 nodes with ~24B tokens.
- Solves spatial forgetting via geometry routing and temporal drifting via self-augmented training.
- Includes interactive GUI, training code, and a 4-step DMD distillation LoRA.
- Weights on Hugging Face under NVIDIA research-only license, code is Apache 2.0.
NVIDIA's Spatial Intelligence Lab has released Project Lyra 2.0, an open generative 3D world model that takes one image and turns it into a walkable scene. The pipeline synthesizes a camera-controlled walkthrough video, then lifts it into an explicit 3D representation you can render, simulate, or drop a robot into. The repo has crossed 2.2k stars, and the model is up on Hugging Face with inference, training code, and an interactive GUI.
One picture, a persistent scene
Lyra 2.0 is built around what the authors call generative reconstruction: generate camera-controlled videos that simulate scene walkthroughs, then lift them to 3D via feed-forward reconstruction. The output is a set of 3D Gaussian Splats, small colored ellipsoids that together form a scene renderable in real time from any viewpoint.
The model takes a single image and synthesizes a long-range video with global geometric consistency using a WAN-14B backbone, then reconstructs that video into an explicit 3D Gaussian representation for real-time rendering. It accepts a 480x832 input image plus a camera trajectory (81 frames recommended) and produces a .ply point cloud of Gaussians, each with position, covariance, color, and opacity.
The two failure modes it fixes
Existing long-horizon video generators fall apart when a camera loops back to look at something it saw earlier. Two forms of degradation dominate: spatial forgetting and temporal drifting. Spatial forgetting happens when older frames slide out of the model's temporal context and it has to hallucinate what used to be there. Temporal drifting is the slower rot, where autoregressive generation accumulates small synthesis errors that gradually distort scene appearance and geometry.