UT Austin's ExpertEdit Rewrites Novice Athletic Motion Into Expert Form Without a Coach
A UT Austin framework rewrites your athletic form to look expert-like, learning only from unpaired expert videos with no paired data or text prompts.
ProTakeaways
- ExpertEdit rewrites a novice athlete's motion into expert-like form using only unpaired expert video for training.
- Uses a VQ-VAE pose tokenizer plus a BERT-style transformer that infills masked spans at kinematic peaks.
- Preserves global translation, root orientation, and body shape; only edits joint rotations at skill-critical moments.
- Beats TMED, SimMotionEdit, and FLAME by 2-4x on FID improvement across basketball, soccer, and karate.
- Reverse layup FID improvement of 12.08% vs 6.13% for the best supervised baseline requiring paired data.
- Paired evaluation benchmark released on Zenodo; paper on arXiv.
Watching yourself perform a layup with the polished form of an NBA player turns out to be one of the fastest ways to actually learn the shot. That psychological quirk, called video self-modeling, is the motivation behind ExpertEdit, a new framework from UT Austin that automatically rewrites a novice's motion capture into an expert-quality version of the same action, without ever needing paired novice-expert examples or text prompts telling it what to fix.
The system, published by Arjun Somayazulu and Kristen Grauman, tackles a problem that has quietly blocked AI coaching: existing motion editors need someone to tell them what to change. ExpertEdit learns it by watching how experts move, then subtly nudges a novice's joints into that manifold at the exact moments where skill actually differs.
Why paired data is a dead end for skill coaching
Text-driven motion editing has become a popular paradigm, with models like TMED, SimMotionEdit, and FLAME taking a source clip plus an instruction such as "raise the right arm by 15 degrees" and producing an edited output. That works for coarse changes like walking to dancing, and falls apart when the goal is to improve the same action rather than change it.
The reason is data. MotionFix, one of the standard paired motion editing datasets, was built by running similarity search to find candidate pairs and then hiring humans to write edit descriptions. Over 55% of candidate pairs were rejected as too similar to label. Scaling that to skill refinement would require domain experts capable of diagnosing subtle execution errors on tens of thousands of clips. At inference time, these systems still need a coach in the loop to write the correction, which defeats the point of an AI coach.
Infill the moments where skill lives
ExpertEdit reframes skill refinement as contextual motion infilling. During training it only ever sees expert clips, learning to reconstruct short masked spans of expert motion from the surrounding context using a masked language modeling objective borrowed from BERT. At inference, it applies the same mask to a novice clip at the skill-critical moment and lets the model fill in what an expert would do there, conditioned on the novice's own surrounding motion.