Skoltech Builds RL Benchmarks Where the Optimal Answer Is Already Known
A new framework builds reinforcement learning benchmarks with mathematically certified optimal policies, giving researchers a true ground-truth reference for comparing algorithms.
ProTakeaways
- Researchers extend converse optimality to stochastic nonlinear systems, generating RL benchmarks with analytically known optimal policies.
- Framework provides necessary and sufficient conditions for a prescribed value function to be optimal under Gaussian noise.
- Two benchmark families released: ConverseArm (n-link robot arms) and NVDEx (unstable unicycle vehicles with dynamic extension).
- Evaluation of PPO, SAC, TD3, DDPG, A2C shows PPO wins on stationary distributions and long horizons.
- Regret paradoxically grows with control authority because stronger control amplifies sensitivity to policy imperfections.
- Code, fixtures, and evaluation scripts available at github.com/converseoptimality/RL-Benchmarking.
Comparing reinforcement learning algorithms is a mess. Scores swing wildly depending on reward shaping, initial state distributions, hyperparameters, and even the random seed. A new paper from researchers at Skoltech, Innopolis, TU Chemnitz, and collaborators flips the usual benchmarking approach on its head. Rather than running algorithms on environments and hoping the numbers mean something, they construct environments where the optimal policy is already known analytically, then measure exactly how far each algorithm falls short.
The technique, called stochastic converse optimality, extends a classical control-theory idea to noisy, discrete-time nonlinear systems. Converse optimality solves the inverse of the usual Bellman problem: given a target optimal value function and stage costs, what system dynamics and feedback laws make Bellman's optimality equation hold globally? The result is a benchmark family where the ground truth is designed in rather than estimated.
Why current RL benchmarks fall apart
The core issue with suites like Procgen, D4RL, or Open RL Benchmark is that they lean on learning curves and cumulative reward tables, and those metrics miss critical design challenges such as environment sensitivity, robustness, and reproducibility. Without a known optimum, algorithms can only be compared relative to each other, and those comparisons are confounded by high variance, protocol choices, and the implementation quirks Henderson et al. famously exposed years ago.