QQL Kills Offline RL's Biggest Tuning Problem Without Sacrificing Performance
A new offline reinforcement learning method eliminates per-dataset temperature tuning in Extreme Q-Learning by deriving it from quantile regression.
Takeaways
- QQL removes the temperature
βhyperparameter from Extreme Q-Learning by estimating it via quantile regression. - Under a Gumbel error assumption,
β(s) = (V̂(s) - V(s))/ω, a difference of two learned value quantiles. - A mild-generalization term with conservative estimation adds controlled optimism beyond the dataset.
- Uses one hyperparameter setting across all D4RL and NeoRL2 tasks tested.
- Matches or exceeds XQL, MXQL, IQL, CQL, TD3+BC on most benchmarks, at 1.6× the wall-clock cost.
- Full paper and derivations on arXiv and OpenReview.
Offline reinforcement learning promises a lot: learn a policy from a fixed batch of logs, never touch the real environment, and deploy safely in domains where interaction is expensive or dangerous. In practice, one of the strongest recent methods in the space, Extreme Q-Learning (XQL), has been held back by a stubborn issue. It works well when you hand-tune a single temperature parameter for every dataset, and falls apart when you don't. A new paper called Quantile Q-Learning (QQL) proposes a way to compute that temperature directly from data using quantile regression, and reports competitive or better results with a single hyperparameter setting across every benchmark it tries.
The temperature problem in Extreme Q-Learning
Extreme Q-Learning models the Bellman error distribution using the Extreme Value Theorem, assuming it follows a Gumbel distribution. That assumption gives a clean, in-sample loss that avoids querying out-of-distribution actions, and it has been competitive on standard benchmarks like D4RL. The catch is the temperature coefficient β that sits inside two exponential terms in the value and policy losses.
Both XQL and its stabilized variant MXQL are sensitive to hyperparameter choices, particularly the temperature coefficient, which often requires dataset-specific tuning to achieve optimal performance. This sensitivity can cause significant performance degradation when hyperparameters are changed. MXQL tried to stabilize training through a Maclaurin-series-based approximation of the original XQL loss, but it still needs manual per-dataset tuning of β, so the underlying problem stays.
To make this concrete, the authors report that the best β for XQL varies from 0.5 on the Adroit hammer-human dataset up to 10.0 on walker2d-medium. Ship a single value and performance collapses on several tasks.