Tsinghua's RoboTAG Trains Robot Pose Estimation on Unlabeled Video
A Harvard and Tsinghua team's dual-branch graph fuses 2D image features with 3D depth priors, learning robot pose from unlabeled video and hitting SOTA on DREAM.
PRO- RoboTAG estimates robot joint angles and camera pose from a single RGB image using a dual 2D/3D branch graph.
- A topological graph with forward and alignment edges enables 2D-3D consistency losses on closed loops, no labels needed.
- Uses Depth Anything V2 Tiny as 3D backbone, allowing training on unannotated in-the-wild robot videos.
- Hits 76.9% average AUC on DREAM, beating Holistic Pose, RoboPEPP and RoboPose by 1.2%.
- Runs inference in 35 ms per frame, versus 200+ ms for render-and-compare methods like RoboPose.
- Weak spot: gripper joints and in-distribution synthetic sets; accepted at CVPR 2026.
Figuring out where every joint of a robot arm sits from a single photo sounds like a solved problem. It isn't. Most methods lean on 2D image backbones and require large piles of labeled training data, which is exactly what real deployments do not have. A team from Tsinghua, Harvard, MIT, NTU and CMU is proposing a fix called RoboTAG, short for Robot Topological Alignment Graph, and it treats the whole camera-plus-robot system as a graph where 2D and 3D views of the same variable have to agree with each other.
The result is a monocular pose estimator that trains from unlabeled in-the-wild robot videos, generalizes better to out-of-distribution scenes, and beats the previous best method by 1.2 points of average AUC on the standard DREAM benchmark.
Why monocular robot pose is stuck
Existing methods build networks on top of 2D visual backbones and depend heavily on labeled data for training, which is often scarce in real-world scenarios, causing a sim-to-real gap. They also flatten an inherently 3D problem into the 2D domain, neglecting 3D priors. Approaches like RoboPose iteratively render and compare a robot mesh against the image, which is accurate but expensive. RoboPEPP predicts 2D keypoints and then runs Perspective-n-Points to solve the camera, meaning any occluded or missed keypoint corrupts the final pose.
What no one had really tried was pulling in a modern pretrained 3D vision model such as Depth Anything and letting it co-train with the 2D branch on unlabeled footage. That is the gap RoboTAG fills.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
