---
title: "ReMAC Brings Best-of-K Exploration to Robot Control Without Entropy Tricks"
subtitle: "ReMAC brings retry-based exploration objectives from LLM training into continuous robot control, matching SAC without any entropy bonus"
url: "https://alphasignal.ai/news/remac-brings-best-of-k-exploration-to-robot-control-without-entropy-tricks"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-04T00:00:00.000Z"
updated: "2026-06-12T01:28:19.524Z"
type: "Paper"
topics:
  - "Reasoning"
  - "Post Training"
subtopics:
  - "Test Time Compute"
  - "Rl"
upvotes: 1190
source_url: "https://arxiv.org/abs/2606.05888"
comments: 0
truncated: true
---

# ReMAC Brings Best-of-K Exploration to Robot Control Without Entropy Tricks

> ReMAC brings retry-based exploration objectives from LLM training into continuous robot control, matching SAC without any entropy bonus

There is a quiet idea gaining traction across both language model training and classical reinforcement learning: instead of asking an agent to get it right on the first try, let it attempt a task multiple times and reward the **best** outcome. This is the logic behind `pass@K` metrics in coding benchmarks and `max@K` objectives in RL. A new paper from Soichiro Nishimori and Paavo Parmas takes this idea and pushes it somewhere it has never gone before -- into the continuous, high-dimensional action spaces used for robot locomotion and control.

## The exploration tax every RL engineer pays

Getting a reinforcement learning agent to explore is one of the oldest unsolved problems in the field. The standard fix is an entropy bonus: you add a term to the loss that explicitly penalizes the agent for being too confident, forcing it to try diverse actions. [Soft Actor-Critic (SAC)](https://arxiv.org/abs/1801.01290), the dominant off-policy algorithm for continuous control, does exactly this. SAC optimizes a stochastic policy using a maximum entropy objective, encouraging exploration by promoting diverse actions, thereby effectively balancing exploration and exploitation. It works well, but it introduces a temperature hyperparameter that must be tuned or automatically adjusted, and the exploration is always explicit -- baked into the objective by hand.

The alternative is to make exploration **emerge** from the objective itself. ReMax proposes an RL objective whereby stochastic exploration arises as an emergent property, without adding any explicit exploration bonus. In ReMax, an episode is modified so that the agent can reset to previous states in the trajectory, and the agent's goal is to maximize the best return in the trajectory tree. The key insight: if you are going to get multiple shots at a task, a deterministic policy that always picks the same action is useless -- you need variance to improve your best-of-K score. Exploration becomes rational, not enforced.

The catch? ReMax was originally formulated for

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Retry Policy Gradients in Continuous Action Spaces (arXiv)](https://arxiv.org/abs/2606.05888)
- [Emergence of Exploration in Policy Gradient RL via Resetting (ReMax, OpenReview)](https://openreview.net/forum?id=GKsNIC_mQRG)
- [Corrected Soft Actor Critic for Continuous Control (arXiv)](https://arxiv.org/html/2410.16739v1)
- [Monte Carlo Gradient Estimation in Machine Learning (arXiv)](https://arxiv.org/pdf/1906.10652)
- [Pass@k Training for Adaptively Balancing Exploration and Exploitation (arXiv)](https://arxiv.org/abs/2508.10751)