Maryland's AFFORD2ACT Teaches Robots to Act Using Just 20 Dots
A University of Maryland team turns robot manipulation into a 38-number game, hitting 82% success on unseen objects with just 15 minutes of training.
Takeaways
- AFFORD2ACT distills a single RGB image and text prompt into 19 semantic 2D keypoints for robot manipulation.
- The 38-dimensional state policy trains in about 15 minutes on a single RTX 2080 GPU.
- Hits 82% success on unseen objects, novel categories, new backgrounds, and distractor scenes across six tasks.
- Uses affordance heatmaps plus DINO features to select functionally meaningful points, then Co-Tracker for temporal tracking.
- A gating network dynamically re-weights keypoints per phase; removing it drops success from 86% to 52%.
- Paper: arxiv.org/abs/2510.01433. Project page: afford2act.github.io.
Robots that learn from pixels tend to overfit their training rooms. Change the background, swap the mug for a teapot, or drop a distracting object into the scene and the policy often falls apart. A team from the University of Maryland has a proposal that goes hard in the opposite direction: throw away almost the entire image and let the robot see the world as fewer than 20 dots.
Their new framework, AFFORD2ACT, distills a single RGB frame plus a text instruction into a tiny set of 2D keypoints anchored to the parts of an object that actually matter for the task. The result is a 38-dimensional state policy that trains in about 15 minutes and runs in real time without proprioception or dense representations. Across six real-world tasks, the approach hits an 82% success rate on unseen objects, novel categories, backgrounds, and distractors.

The dense-input tax
Behavior cloning from raw RGB or RGB-D has become the default recipe for visuomotor policies, and it comes with baggage. High-dimensional pixel inputs entangle background, lighting, and instance appearance with the tiny sliver of information the robot actually needs, and they demand large datasets to disambiguate.
Keypoint-based methods promise a way out, but picking which points to track is hard. Prior work has leaned on three imperfect options: motion-based selection that breaks when the background moves, end-to-end learned keypoints that get tangled up with the training set, or hand-labeled points that do not scale. AFFORD2ACT takes a fourth route by using language-conditioned affordance models, which predict where in an image a given action like