World Labs Drops 3 Papers Turning Flat Images Into Full 3D Worlds

World Labs, the spatial intelligence company co-founded by Fei-Fei Li, has published three research papers in a single drop, each tackling a different angle of the same hard problem: how do you get rich, complete 3D geometry out of 2D inputs? The trio covers depth prediction from a single image, dynamic 4D reconstruction from monocular video, and a unified model for joint text-image-depth reasoning. Together, they sketch out a research agenda for building world models that actually understand space.

Why this is hard

Standard depth estimation gives you one depth value per pixel -- a flat sheet draped over the visible surface of a scene. That works fine for what you can see, but the moment something is occluded (hidden behind another object), you get nothing. Reconstructing dynamic scenes from a single moving camera is even harder: you have no second viewpoint to triangulate from, no known camera parameters, and the scene itself is changing. The field has made steady progress, but most methods still require multi-view setups, calibrated cameras, or static scenes to work reliably.

World Labs is betting that large-scale generative models -- the same diffusion and video models powering image and video synthesis -- carry enough implicit 3D knowledge to close that gap. All three papers are built on that thesis.

Paper 1: World Tracing -- depth as a stack, not a sheet

The first paper introduces World Tracing, a method that predicts full 3D geometry from a single image, including surfaces that are completely hidden from view. The key idea is to stop thinking about depth as a single value per pixel and instead predict a stack of depth values for each pixel -- essentially peeling the world into layers, one behind the other.

• Each pixel gets multiple depth predictions, not just the nearest visible surface
• A diffusion model generates this layered depth representation, letting it hallucinate plausible geometry behind occluded regions
• The output stays faithful to the visible image while filling in what cannot be seen

This is a meaningful departure from conventional monocular depth estimation. By framing depth as a generative problem over a distribution of layers rather than a regression to a single value, the model can reason about what is likely behind a wall, under a table, or around a corner. The diffusion backbone is what makes this tractable -- it has seen enough of the world during pretraining to make educated guesses about hidden geometry.

Paper 2: Flex4DHuman -- dancing into 4D

The second paper, Flex4DHuman, targets a specific and practically important case: lifting a monocular video of a person into a fully dynamic 4D representation. 4D Gaussians (short for 4D Gaussian Splatting) are a way to represent a scene as a cloud of small, fuzzy ellipsoids that move and deform over time -- think of it as a point cloud that has been given both shape and a timeline.

The pipeline works in two stages:

1. A video diffusion model is fine-tuned to generate synchronized multi-view videos from the original monocular input -- essentially imagining what the scene would look like from multiple cameras at once
2. Those synthetic multi-view videos are then distilled into 4D Gaussians, producing a representation you can render from any angle at any point in time

The payoff is striking: a video of a person dancing can be lifted into a 4D asset and composited directly into a 3D world. That is the kind of pipeline that game studios, VFX houses, and virtual production teams have needed for years -- one that does not require a multi-camera rig or a controlled studio environment.

Paper 3: Modality Forcing -- one model, three tasks

The third paper, Modality Forcing, is the most architecturally interesting of the three. The central claim is that text-to-image generation is a scalable pretraining objective for 3D reasoning -- and that you can adapt an existing text-to-image model to jointly understand text, RGB images, and depth maps without training a new model from scratch.

The technique works by forcing the model to handle different input-output combinations during training: sometimes it gets text and must output an image, sometimes it gets an image and must output depth, sometimes it gets depth and must output an image. This multi-task pressure teaches the model a shared internal representation that spans all three modalities.

• Text-to-RGBD: generate a color image and its depth map together from a text prompt
• Depth estimation: predict depth from an RGB image
• Depth-to-image: synthesize a plausible image given a depth map

Unifying these three tasks in a single model is not just elegant -- it is efficient. You get a depth estimator, a conditional image generator, and a joint RGBD synthesizer for the cost of one fine-tuning run on top of an existing text-to-image backbone.

The bigger picture

What ties all three papers together is the use of 2D generative priors -- knowledge baked into large image and video models -- as a scaffold for 3D understanding. Rather than building 3D-specific architectures from scratch, World Labs is showing that the representations already learned by diffusion models are rich enough to support geometry prediction, dynamic reconstruction, and multi-modal 3D reasoning.

World Labs has been building toward this since launching Marble, its flagship 3D world generation product. These three papers represent the research layer underneath that product: the methods that let a model go from a flat image or a phone video to a navigable, editable 3D scene. The practical targets are clear -- game asset creation, VFX, virtual production, robotics simulation, and spatial AI applications that need geometry, not just pixels.

All three projects were led by interns at World Labs. No code or model weights have been released alongside the papers at time of writing, but the thread links to the individual research pages for each project.