Brown University's DyTact Captures Exact Hand-Object Contact With 30x Fewer Gaussians
A new markerless system uses 2D Gaussian surfels rigged to MANO meshes to reconstruct dynamic hand-object contacts with 10k Gaussians instead of 300k.
Takeaways
- DyTact captures dynamic hand-object contacts from multi-view video using 2D Gaussian surfels bound to MANO meshes (paper).
- Beats MANUS, HARP, and MANO baselines on the new DyTact-21 benchmark with 0.226 mIoU and 0.378 F1.
- Uses roughly 10k Gaussians per sequence versus 300k for prior Gaussian-based hand models.
- An MLP refinement module predicts per-frame offsets to handle skin deformation near contact regions.
- Contact-guided adaptive density control voxelizes the scene and concentrates isotropic surfels in occluded contact areas.
- Accepted as a 3DV 2026 Oral; code available on GitHub.
Capturing what happens when a hand actually touches an object, the exact patches of skin that press, slide, and release, is one of those problems that sounds solved until you look closely. Instrumented gloves get in the way. Thermal cameras only see accumulated heat. Parametric hand templates, which most researchers rely on, drift noticeably away from real hand geometry, making contact estimation unreliable exactly where it matters most.
A team from Brown University and IIT Delhi has put out DyTact, a markerless capture method that reconstructs dynamic hand-object contacts from multi-view video. The paper was accepted as a 3DV 2026 Oral, and the code is on GitHub.
The misalignment problem nobody wants to talk about
MANO is the standard parametric hand model in this space. MANO-based approaches generalize well across scenarios but frequently suffer from noticeable misalignments between the parametric template and the actual hand geometry, which undermines fine-grained contact estimation. The root cause: MANO provides a generic, low-dimensional representation of shape, while accurate contact estimation requires a personalized one.
Training a neural hand avatar per person works but is slow and requires huge amounts of data. Neither option is great if you want scalable, natural, high-fidelity contact capture.

2D surfels, glued to the mesh
DyTact's core idea combines the best of both worlds. It uses multi-view markerless capture and represents hands and objects with 2D Gaussian surfels that accurately model surfaces and appearance, enabling precise contact analysis while inherently avoiding misalignment issues. A surfel here is a flat, oriented 2D Gaussian disk that hugs the surface rather than a fuzzy 3D blob, which matters when you care about where two surfaces actually meet.