Skip to main content
Back

University of Antwerp's Robust MPC Shields RL Robots Without Retraining

A new method adds adversarial robustness to any pre-trained RL agent at inference time, no retraining required, using model-predictive control and worst-case rollouts.

Paper, Full Access
  • Post-hoc robustness: A new method adds adversarial robustness to any pre-trained model-based RL agent at inference time, with no retraining required.
  • How it works: Uses model-predictive control (MPC) with adversarial rollouts -- simulating worst-case environment shifts via projected gradient descent inside a bounded uncertainty set.
  • OOD handling: Explicitly detects and mitigates out-of-distribution states during planning, preventing the world model from giving unreliable predictions under heavy perturbation.
  • Benchmarks: Validated on Gymnasium MuJoCo Reacher-v4 and Hopper-v4 with perturbations to damping, gear ratios, torso mass, and friction -- realistic sim-to-real shift scenarios.
  • Results: Robust MPC consistently outperformed the nominal baseline policy under all tested perturbation types, with the adversarial planning component confirmed as the key driver of improvement.
  • Limitations: Adds per-step compute cost; requires an existing world model; degrades if the deployment environment is severely out-of-distribution from training. Read the paper.

Training a reinforcement learning agent that actually works in the real world is harder than it looks. A policy that scores perfectly in simulation can collapse the moment it encounters a slightly different friction coefficient, a heavier limb, or a worn-out motor. This fragility is one of the biggest blockers to deploying RL in robotics and control. A new paper from researchers at the University of Antwerp tackles this problem from a fresh angle: instead of making training more robust, what if you could bolt robustness onto an already-trained agent, at inference time, without touching the neural network at all?

That is the core idea behind Post-Hoc Robustness for Model-Based Reinforcement Learning. The method, called Robust MPC (model-predictive control), wraps around any pre-trained policy and a learned world model to produce safer, more conservative actions when the environment drifts from what the agent was trained on.

Why robustness is so hard to get right

Real-world RL applications are often hindered by the absence of robustness in learned policies. An RL agent that trains in a certain environment often struggles to perform well in nearly identical ones. This is the sim-to-real gap in practice: the simulation-to-reality gap and inherent variability in real control systems limit the adoption of RL in real-world scenarios.

The standard fix is to bake robustness into training. Adversarial training in RL improves robustness by repeatedly exposing the agent to adversarial examples during training, similar to inoculating it against real-world attacks. But this requires re-running the full training loop, which is expensive, and it locks robustness to whatever perturbations you anticipated at training time. Neural network policies are vulnerable to carefully crafted perturbations to their inputs, and these problems are even more severe in RL than in supervised learning, as the effects of successive mistakes can cascade over a long time horizon.