Axolotl3D completes 3D objects from partial views

NVIDIA’s Spatial Intelligence Lab has introduced Axolotl3D, a research model that reconstructs object geometry from incomplete images and point samples. Its unified conditioning system supports single images, sparse multi-view captures, occluded objects, and targeted shape edits with one diffusion model.

Image-to-3D generators such as Hunyuan3D can produce detailed meshes from a photograph, but hidden surfaces remain unconstrained. Occlusion and inconsistent camera estimates compound the problem across multiple views. Axolotl3D adds explicit geometric evidence, visibility information, and camera parameters so observed surfaces remain aligned as the model predicts missing regions.

Partial evidence becomes a constraint

Axolotl3D accepts combinations of posed images, visibility masks, camera parameters, and a partial point cloud. Each input contributes a different constraint:

The generator operates on a compressed representation called a shape latent. Diffusion progressively refines that latent under guidance from the available inputs, using patterns learned from training meshes to infer unobserved surfaces. A decoder then converts the result into completed geometry.

From pixels to condition tokens

Axolotl3D extends Hunyuan3D-DiT, a diffusion transformer paired with a ShapeVAE decoder. Separate encoders translate images and point clouds into tokens that the transformer can process together.

• Image encoding: DINOv2 extracts semantic and visual features from each posed view.
• Geometry encoding: VecSetX represents the observed point cloud.
• Feature fusion: Feature-pyramid layers combine the encoded inputs into multimodal condition tokens.
• Cross-attention: Hunyuan3D-DiT consults those tokens while generating the shape latent.
• Geometry decoding: ShapeVAE converts the generated latent into a completed 3D object.

Training on deliberate blind spots

During training, the pipeline derives varied conditioning examples from large 3D mesh collections. It changes the number of views, masks visible regions, samples partial geometry, and varies which modalities are available. Repeated exposure to these missing-data patterns allows one checkpoint to handle several reconstruction and editing tasks.

This training design also explains the role of the point cloud. Image features help identify an object and suggest its likely structure; sampled geometry supplies direct spatial evidence. Camera calibration keeps both forms of evidence aligned across views.

What the benchmarks support

The paper evaluates Axolotl3D on Toys4K and OmniObject3D under clean and synthetically occluded conditions. The authors report leading results across those settings, along with experiments in real-world reconstruction and geometry-guided editing. Exact metric definitions, baseline configurations, and numerical comparisons appear in the paper’s evaluation tables.

Qualitative examples include bicycles, chairs, animals, and household objects with substantial portions hidden. Their recovered surfaces reflect regularities learned from related training shapes. Those concealed surfaces remain model predictions, so benchmark performance does not guarantee physical accuracy for an individual object.

Three supported workflows

1. Sparse-view image-to-3D: Axolotl3D can consume camera and point predictions from Pi3X to generate a mesh from one or a few photographs. The experiments indicate tolerance for imperfect predicted point clouds.
2. Geometry-guided editing: An edited or inpainted image defines the requested change, and conditioning points anchor unaffected regions. This reduces unintended geometry changes outside the edited area.
3. Simulation asset preparation: Completing partially captured objects can produce fuller collision geometry for robotics and synthetic-data pipelines. Topology, scale, and inferred surfaces still require validation before physical simulation.

Deployment boundaries

Axolotl3D targets object-level reconstruction and is evaluated primarily on discrete items such as toys and household objects. It does not reconstruct complete rooms or outdoor scenes. Practical use also requires camera poses or an upstream estimator, suitable object observations, and checks for errors in surfaces that were never visible.

As of the linked project materials, NVIDIA has not announced a public checkpoint or source-code release. The demonstration uses Kaolin’s web UI, but that integration does not establish a release plan or production API.

A reusable conditioning pattern

For teams working with Hunyuan3D-style diffusion transformers, Axolotl3D presents a practical architecture pattern: encode each evidence source separately, align geometric inputs in one coordinate frame, fuse them as condition tokens, and train one generator across varied observation regimes. Reproducing the approach still requires mesh-scale training data, synthetic partial-observation generation, camera handling, and a compatible shape decoder.