Skip to main content
Back

SFU's EgoFun3D Turns Phone Videos Into Robot Simulator-Ready 3D Objects

A new dataset, task, and pipeline turn egocentric video into simulation-ready 3D objects where one part actually drives another.

Pro

  • EgoFun3D turns egocentric videos into simulation-ready interactive 3D objects with cross-part functional couplings.
  • Function templates split objects into receptor and effector, with four physical effects and four mapping types.
  • Templates compile to executable Python for Genesis, Isaac Sim, and BEHAVIOR-1K simulators.
  • Dataset contains 271 videos with 2D/3D segmentation, articulation, and function template annotations.
  • Gemini 3 Flash hits 92.9% accuracy on function template inference; articulation estimation remains the biggest bottleneck.
  • Code, dataset, and Gradio demo publicly available from the Simon Fraser University 3DLG lab.

Building interactive 3D objects for robotics and embodied AI simulators is painful. You usually need synthetic assets, hand-authored URDFs, or a lab full of scanning gear. A new project from Simon Fraser University proposes a different route: watch a first-person video of someone using an object, then automatically produce a simulator-ready asset where turning a knob actually changes a burner's temperature.

The paper introduces EgoFun3D, which bundles a task formulation, a 271-video dataset, and a four-stage baseline pipeline. The goal is to obtain simulation-ready interactive 3D objects from egocentric video input, motivated by the fact that interactive assets are in high demand for embodied AI yet scarce, making it valuable to model them from readily available real-world footage.

Modeling what a part actually does

Most prior work on interactive 3D reconstruction focuses on articulation, meaning how a hinge rotates or a drawer slides. That misses the point of many household objects. Turning a stove knob is not interesting because the knob rotates. It is interesting because the burner heats up.

EgoFun3D captures these cross-part functional mappings (e.g., rotation of a stove knob controls burner temperature) through function templates, a structured computational representation. A function template splits an object into a