Skip to main content
Back

TU Darmstadt's 3DRealHead Builds Realistic 3D Avatars From Just 3 Photos

A new head avatar pipeline reconstructs a driveable 3D face from 1 to 3 selfies, then animates the mouth using raw video gradients instead of relying purely on 3DMM parameters.

Pro

  • 3DRealHead reconstructs a driveable 3D head avatar from just 1-3 selfies via few-shot inversion of a learned prior.
  • A Style U-Net emits 3D Gaussians attached to a FLAME mesh, rendered at 27 fps.
  • Mouth region is driven by image gradient features from the driving video, bypassing 3DMM limitations.
  • Trained on NeRSemble (200 subjects, 16 views, 5 days on L40S).
  • Beats SynShot and rivals per-identity methods trained on 3000+ frames on in-the-wild data.
  • No public code or weights; built for self-reenactment, struggles with cross-reenactment and extreme views.

Building a convincing 3D avatar of yourself has traditionally required either a multi-camera studio or minutes of dedicated video footage. A team from TU Darmstadt and Google proposes something lighter: snap one to three photos, plug in a webcam, and drive a full 3D head avatar in real time. Their method, 3DRealHead, focuses on the region most systems botch, the mouth interior.

Existing 3D head avatar methods often fail to reproduce identity and facial expressions even with multi-view data or learned priors, and the mouth and teeth region is particularly hard because training data for those person-specific features is scarce. Most pipelines also drive expressions through a 3D Morphable Model (3DMM), a low-dimensional statistical face model. That constrains the avatar to whatever the 3DMM was designed to represent, which rarely goes beyond broad jaw and eye movements.

Teaser showing few-shot inputs on the left producing a driveable 3D head avatar on the right

Sidestepping the 3DMM straitjacket

The core trick is a hybrid expression signal. The 3DMM still handles coarse motion (jaw pose, general expression), but for the mouth region the model reads the actual driving video and extracts image gradient features. Avatar reconstruction runs through a few-shot inversion of a 3D human head prior, represented as a Style U-Net that emits 3D Gaussian primitives which can be rendered from novel views. Those gradients tell the network what the inside of the mouth is doing frame by frame, so tongue positions and teeth articulation that would collapse into a blurry hole under a pure 3DMM approach get preserved.