Skip to main content
Back

Maryland's AFFORD2ACT Teaches Robots to Act Using Just 20 Dots

A University of Maryland team turns robot manipulation into a 38-number game, hitting 82% success on unseen objects with just 15 minutes of training.

Paper, Full Access
  • AFFORD2ACT distills a single RGB image and text prompt into 19 semantic 2D keypoints for robot manipulation.
  • The 38-dimensional state policy trains in about 15 minutes on a single RTX 2080 GPU.
  • Hits 82% success on unseen objects, novel categories, new backgrounds, and distractor scenes across six tasks.
  • Uses affordance heatmaps plus DINO features to select functionally meaningful points, then Co-Tracker for temporal tracking.
  • A gating network dynamically re-weights keypoints per phase; removing it drops success from 86% to 52%.
  • Paper: arxiv.org/abs/2510.01433. Project page: afford2act.github.io.

Robots that learn from pixels tend to overfit their training rooms. Change the background, swap the mug for a teapot, or drop a distracting object into the scene and the policy often falls apart. A team from the University of Maryland has a proposal that goes hard in the opposite direction: throw away almost the entire image and let the robot see the world as fewer than 20 dots.

Their new framework, AFFORD2ACT, distills a single RGB frame plus a text instruction into a tiny set of 2D keypoints anchored to the parts of an object that actually matter for the task. The result is a 38-dimensional state policy that trains in about 15 minutes and runs in real time without proprioception or dense representations. Across six real-world tasks, the approach hits an 82% success rate on unseen objects, novel categories, backgrounds, and distractors.

Comparison of dense image input versus sparse affordance-guided keypoint input on a mug manipulation task, plus generalization examples.

The dense-input tax

Behavior cloning from raw RGB or RGB-D has become the default recipe for visuomotor policies, and it comes with baggage. High-dimensional pixel inputs entangle background, lighting, and instance appearance with the tiny sliver of information the robot actually needs, and they demand large datasets to disambiguate.

Keypoint-based methods promise a way out, but picking which points to track is hard. Prior work has leaned on three imperfect options: motion-based selection that breaks when the background moves, end-to-end learned keypoints that get tangled up with the training set, or hand-labeled points that do not scale. AFFORD2ACT takes a fourth route by using language-conditioned affordance models, which predict where in an image a given action like