Multi-Task RL Lets Robot AI Generalize to Unseen Objects Without Forgetting
A new recipe for post-training robot foundation models shows value-based RL can improve many tasks at once, and even transfer to unseen ones.
- New paper shows value-based RL post-training can improve generalist robot policies across many tasks at once.
- Critic-guided methods (BoN, OGPO) generalize to unseen scenes and objects; filtered SFT forgets.
- OGPO wins overall by combining off-policy critic learning with PPO-style policy extraction.
- Scaling the number of training objects is the critical axis for transferring to unseen objects.
- Multi-task post-training matches per-task experts in far fewer gradient steps.
- Paper on arXiv; evaluated on MolmoSpaces and LIBERO benchmarks.
Generalist vision-language-action (VLA) models arrive pretrained on large collections of robot demonstrations, yet deployment in a new kitchen or lab often requires additional on-robot training. Fine-tuning a separate policy for each task consumes data and compute, and narrow updates can erase capabilities learned during pretraining. A new study reports that value-based reinforcement learning across a diverse task set can improve performance on training tasks and transfer those gains to unseen scenes and objects.
The paper, Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement, compares three post-training methods while varying the number and diversity of reinforcement-learning tasks. Critic-guided policy extraction through off-policy generative policy optimization (OGPO) produced the strongest overall results. Best-of-N sampling (BoN) showed similar scaling trends with task diversity, although its success rates remained lower.
Narrow tuning erodes transfer
Single-task reinforcement learning concentrates the policy on one deployment objective, which can reduce performance elsewhere. The study instead tests generalist improvement: raising success rates across multiple training tasks while preserving or improving performance on tasks excluded from reinforcement learning.
Task diversity determined whether those gains transferred. As the training set expanded, critic-based methods improved on familiar tasks, new scenes containing familiar objects, and tasks involving unseen objects. Filtered supervised fine-tuning did not follow the same scaling pattern and performed poorly on held-out objects.
Critics unlock the scaling curve
All three methods begin with policy rollouts, but they use the resulting experience differently. A critic is a learned function that estimates the likely return or success of a candidate action. It can guide action selection at inference time or supply a training signal for updating the policy.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
