Skip to main content
Back

CMU's ModAR Beats a 6B Robot Model Using 200 Times Fewer Parameters

A 30M parameter robotics model trained from scratch beats a 6B video-pretrained baseline by predicting depth, tracks, and DINO features before actions.

Paper, Full Access
  • ModAR autoregressively denoises point tracks, DINO features, depth, then RGB before predicting robot actions.
  • A 30.1M parameter model trained from scratch hits 75% success on RoboTwin, beating a 6B Flex-π baseline at 72%.
  • Uses roughly 200x fewer parameters and 20x fewer training FLOPs than the video-pretrained competitor.
  • Ablations show RGB prediction contributes least; tracks, DINO, and depth are the load-bearing modalities.
  • Context noise during training and structured-first generation order are both critical design choices.
  • On real bimanual tasks, ModAR improves further when trained on human video from EgoDex.

Carnegie Mellon researchers have introduced ModAR, a world-action model that predicts future depth, motion, and visual features before generating robot actions. On six simulated manipulation tasks, the 30.1-million-parameter model achieved a 75% average success rate at the 250-demonstration scale, compared with 72% for a 6-billion-parameter system initialized from a pretrained video generator.

The result suggests that robot policies can learn useful world models without spending most of their capacity on RGB pixels. ModAR trains from scratch, uses about 200 times fewer parameters than the larger baseline, and requires roughly 20 times fewer training FLOPs.

Why RGB consumes so much capacity

World-action models jointly predict a future observation and the robot action that should follow. Approaches derived from video generation typically represent that future as RGB frames, as described in earlier WAM work. Those frames contain task-relevant information, but they also encode lighting, texture, and background details that may have little effect on a grasp or handoff.

ModAR instead uses several representations, each capturing a different part of the scene:

  • Point tracks record how selected image coordinates move across frames.
  • DINO features are compact embeddings from a pretrained visual encoder that capture objects and other semantic structure.
  • Depth maps describe scene geometry and distance from the camera.
  • RGB frames preserve full appearance information.

These modalities reduce the burden on the policy by making motion, semantics, and geometry explicit. The remaining design problem is how to combine them without forcing one network pass to resolve every signal at once.

Structure first, actions last