Skip to main content
Back

RoboMorph Gets Beaten by Diffusion Models on Robot Arm Dynamics

Researchers pit diffusion-based sequence models against a Transformer baseline for robot dynamics prediction, finding diffusion far more robust to distribution shift.

Paper, Full Access
  • Paper reframes robot system identification as in-context meta-learning and benchmarks diffusion vs Transformer sequence models.
  • Three diffusion variants tested: inpainting Diffuser, conditioned CNN, conditioned Transformer, against RoboMorph baseline.
  • Diffusion models substantially more robust under distribution shift; inpainting Diffuser achieves best overall accuracy.
  • Warm-started sampling cuts diffusion inference to ~40ms using 5 denoising steps for real-time control.
  • Trained on 3.5M randomized Franka Panda trajectories simulated in parallel via IsaacGym.
  • CNN diffusion yields smoother trajectories; Transformer diffusion better preserves accuracy under aggressive warm-starting.

Predicting how a robot arm will respond to a torque command sounds like a solved problem, but it isn't. Physics-based models miss friction quirks and unmodeled couplings, while learned black-box models tend to fall apart the moment the robot encounters a signal outside its training distribution. A new paper from IDSIA-SUPSI and Politecnico di Milano takes a fresh swing by casting system identification as in-context meta-learning and asking a pointed question: are diffusion models actually better at forward dynamics than a strong Transformer baseline?

Their answer is yes, at least when the test conditions drift away from what the model saw during training.

Learning to identify systems, not fit them

Traditional black-box dynamics models learn a single system. The meta-learning reframing trains one model over an entire family of dynamical systems and then adapts on the fly by looking at a short trajectory of past inputs and outputs. Rather than optimizing a separate neural network for every distinct plant, the meta-model learns the governing rules of entire classes of systems from contextual input-output trajectories. At inference time the model sees a context window of control commands and observations, then predicts how the robot will respond to future inputs.

The deterministic baseline here is RoboMorph, an encoder-decoder Transformer that regresses a single trajectory point estimate. That works fine in-distribution, but the authors argue it has no way to represent uncertainty and collapses badly when the test signal shifts.