Skip to main content
Back

ReMAC Brings Best-of-K Exploration to Robot Control Without Entropy Tricks

ReMAC brings retry-based exploration objectives from LLM training into continuous robot control, matching SAC without any entropy bonus

Paper, Full Access
  • New algorithm ReMAC extends the ReMax retry objective to continuous action spaces using a pathwise derivative estimator.
  • No entropy bonus needed: ReMAC achieves higher policy entropy than SAC without any explicit entropy regularization term.
  • Gradient analysis: ReMax biases gradients toward higher entropy and damps their magnitude -- Adam's epsilon parameter can mitigate the damping.
  • Competitive performance: ReMAC matches SAC across six MuJoCo locomotion benchmarks (Ant, HalfCheetah, Hopper, Walker2d, Reacher, Swimmer).
  • Retry framing: Optimizes the best return across M parallel attempts per update, connecting to pass@K ideas from LLM training.
  • No code released yet; paper available on arXiv with full theoretical derivations.

There is a quiet idea gaining traction across both language model training and classical reinforcement learning: instead of asking an agent to get it right on the first try, let it attempt a task multiple times and reward the best outcome. This is the logic behind pass@K metrics in coding benchmarks and max@K objectives in RL. A new paper from Soichiro Nishimori and Paavo Parmas takes this idea and pushes it somewhere it has never gone before -- into the continuous, high-dimensional action spaces used for robot locomotion and control.

The exploration tax every RL engineer pays

Getting a reinforcement learning agent to explore is one of the oldest unsolved problems in the field. The standard fix is an entropy bonus: you add a term to the loss that explicitly penalizes the agent for being too confident, forcing it to try diverse actions. Soft Actor-Critic (SAC), the dominant off-policy algorithm for continuous control, does exactly this. SAC optimizes a stochastic policy using a maximum entropy objective, encouraging exploration by promoting diverse actions, thereby effectively balancing exploration and exploitation. It works well, but it introduces a temperature hyperparameter that must be tuned or automatically adjusted, and the exploration is always explicit -- baked into the objective by hand.

The alternative is to make exploration emerge from the objective itself. ReMax proposes an RL objective whereby stochastic exploration arises as an emergent property, without adding any explicit exploration bonus. In ReMax, an episode is modified so that the agent can reset to previous states in the trajectory, and the agent's goal is to maximize the best return in the trajectory tree. The key insight: if you are going to get multiple shots at a task, a deterministic policy that always picks the same action is useless -- you need variance to improve your best-of-K score. Exploration becomes rational, not enforced.

The catch? ReMax was originally formulated for