UC Berkeley's RAPO Stops Robot AI From Crashing in the Real World
Berkeley researchers introduce RAPO, a dual formulation robust RL method that steers rollouts toward worst-case dynamics without collapsing into over-conservatism.
ProTakeaways
- RAPO is a robust RL method built on the dual of the KL-robust MDP formulation from Berkeley Hybrid Robotics.
- An adversarial network (AdvNet) learns the dual temperature η per state-action, amortizing expensive root-finding.
- Boltzmann reweighting biases ensemble sampling toward environments where the current policy is most vulnerable.
- Outperforms PPO, PPO+DR, RARL, and EPOpt across Walker2d, Ant, HalfCheetah, and Hopper OOD sweeps.
- On quadrotor payload tracking OOD, RAPO achieves 9.4 cm RMSE and zero crashes versus PPO's 40% crash rate.
- Approximation errors in η enter only quadratically, preserving the γ-contraction of the robust Bellman operator.
Reinforcement learning policies have a persistent embarrassment: they crush their benchmarks in simulation and then fall apart the moment friction, mass, or actuator dynamics drift even slightly from training. A new paper from UC Berkeley's Hybrid Robotics group, Robust Adversarial Policy Optimization Under Dynamics Uncertainty, proposes a cleaner mathematical route around this problem by rebuilding robust RL around its dual formulation rather than hand-crafted adversaries.
The method, called RAPO, combines two ideas that operate at different scales: a learned adversarial network that tilts individual trajectories toward worst-case outcomes, and a Boltzmann reweighting scheme that biases sampling toward the specific ensemble environments where the current policy is weakest. The authors show it holds up on Walker2d, Ant, HalfCheetah, Hopper, and a quadrotor payload-tracking task with zero crashes under out-of-distribution dynamics.
Why sim-trained policies keep breaking
Policies trained in simulation often fail when transferred to real systems due to aleatoric and epistemic uncertainty, where small deviations can accumulate into catastrophic failures. Existing fixes each carry known failure modes. Domain randomization spreads training across perturbations but treats them all equally, producing inefficient or overly cautious behavior. Pure adversarial perturbations emphasize worst cases while lacking distributional grounding and stability. Ensemble methods capture variability but underestimate risk without adversarial reweighting.
The formal remedy is distributionally robust RL, which frames the problem as a robust Markov Decision Process (RMDP): assume the true transition kernel lives inside a KL-divergence ball around the nominal training model, then optimize for the worst kernel in that ball. The trouble is that the primal version of this problem is infinite-dimensional and intractable, so prior work approximates it by explicitly sampling perturbed environments or introducing an adversarial agent, which leaves coverage blind spots.