TU Delft's FMER Trains Robot Policies 7x Faster on Sparse Tasks
FMER swaps diffusion for flow matching in online RL, delivering tractable entropy, mirror descent updates, and up to 7x faster training.
- FMER replaces SDE-based diffusion policies with an ODE flow matching policy for online RL, published at NeurIPS 2026.
- Weighted conditional flow matching loss is proven to be a simulation-free surrogate for policy mirror descent.
- Closed-form entropy term with tanh correction enables principled max-entropy exploration without heuristics.
- On FrankaKitchen 7-task: 40.3% completion vs 23.1% for best diffusion baseline (DIPO).
- Training is 7x faster than QVPO and 10 to 15% faster than efficient diffusion variants.
- JAX code and baselines available at github.com/gtanaelle/FMER.
Online reinforcement learning updates a policy from newly collected interactions, making exploration and stable policy updates central to training. Diffusion policies can represent complex, multimodal action distributions, but their iterative samplers create two costs: the policy entropy is difficult to compute, and end-to-end optimization may require gradients through a long denoising chain. A TU Delft paper titled Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints, available as the FMER paper, replaces the stochastic reverse process with an ordinary differential equation, or ODE, based flow policy.
FMER learns a time-dependent velocity field that transports base noise toward an action distribution. The resulting policy retains the capacity to model multiple action modes while providing a tractable entropy objective and a policy update that avoids backpropagation through the ODE solver. Sampling still requires numerical integration, so the main savings apply to training rather than every part of the pipeline.
Why denoising complicates online RL
Diffusion policies generate actions by repeatedly removing noise. That iterative process can capture behaviors such as reaching for an object with either hand, where a unimodal Gaussian policy may average incompatible actions or settle on one mode. Methods that optimize the sampler end to end, however, can incur the memory and numerical costs of differentiating through every denoising step.
The final action density produced by a stochastic reverse process also lacks a cheap closed-form entropy. Entropy regularization normally rewards a policy for maintaining a broad action distribution, which supports exploration and delays premature concentration. Diffusion-based reinforcement-learning methods often approximate that objective with heuristics, auxiliary samplers, or other indirect mechanisms.
Flow matching starts from noise as well, but learns a velocity field v(x,t,s) that defines an ODE conditioned on state s. Once the initial noise is sampled, the ODE trajectory is deterministic. FMER operates in an unbounded latent action space and applies
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
