Princeton and UC Berkeley's Value Forcing Beats Flow Critics Across 26 Tasks
A new flow-matching method enforces Bellman consistency at every noise level, producing accurate return distributions and stronger offline RL from limited data.
- Value Forcing trains flow-matching critics by enforcing Bellman consistency at every noise level, not just at the endpoint.
- A new flow Bellman equation adds a Gaussian correction term so the equation holds from pure noise to clean returns.
- Prior flow critics (FlowCritic, Value Flows, PCBF) collapse return distributions into narrow peaks.
- Two algorithms released: VF (explicit actor-critic with rejection sampling) and IVF (implicit, expectile-weighted).
- Beats baselines on 26 OGBench and OCBench tasks, with largest gains at 10 to 25 percent dataset size.
- Code and project page available at github.com/chongyi-zheng/value-forcing.
Value functions remain difficult to estimate when demonstrations are limited and rewards arrive late. Researchers at Princeton and UC Berkeley address that failure in the Value Forcing paper, which introduces a Bellman objective for flow-based critics and reports gains across 26 long-horizon offline reinforcement learning tasks.
Small datasets amplify bootstrap errors
An offline RL transition records an immediate reward, while the total future return remains unobserved. Temporal-difference methods estimate that return from their own predictions, allowing early errors to propagate through long trajectories. Sparse rewards and limited data leave few observations to correct those errors.
Scalar critics also discard information by compressing every possible return into an average. An action that produces either -1 or +1 with equal probability has the same mean as an action that always produces 0, despite carrying a different outcome distribution.
Distributional RL preserves that uncertainty by modeling the full return distribution. Recent critics have used flow matching, a generative method that learns a velocity field for transforming Gaussian noise into return samples through an ordinary differential equation, or ODE.
The noise schedule exposes the failure
The paper traces a recurring problem in earlier flow critics to their Bellman supervision. Their learned return distributions often contract into narrow peaks, erasing uncertainty and behaving much like scalar estimates.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
