Skip to main content
Back

ICQL Beats Every Offline RL Baseline by Learning Q-Functions on the Fly

A new offline reinforcement learning framework replaces the single global Q-function with locally inferred estimates from retrieved transitions, boosting scores across MuJoCo, Adroit, and Kitchen benchmarks.

Pro

Paper, Full Access
  • ICQL replaces global Q-functions with local ones inferred in-context from retrieved neighbor transitions.
  • Linear Transformer layers implement TD updates internally, provably approximating local weight vectors.
  • Gains of 16.4% on Kitchen, 8.8% on MuJoCo, 6.3% on Adroit versus strongest baselines.
  • Q-estimate similarity with online SAC oracle: 0.69 for ICQL versus 0.29 for IQL on Walker2d-Medium.
  • Fails on Hammer-Human due to sparse data and low-quality trajectories breaking locality assumptions.
  • Method is architecture-agnostic and integrates into IQL, TD3+BC, and other actor-critic offline RL frameworks.

Offline reinforcement learning has a stubborn problem: it tries to squeeze one global value function out of a static dataset that often contains wildly different behaviors. A new method called In-Context Compositional Q-Learning (ICQL) takes a different route, using a linear Transformer to infer a fresh, local Q-function on the fly from a handful of retrieved neighbor transitions rather than fitting a single Q across the entire state space.

On D4RL benchmarks, ICQL delivers score gains of up to 16.4% on Kitchen tasks, 8.8% on MuJoCo, and 6.3% on Adroit compared to the strongest prior baselines.

Why one Q-function buckles

The standard offline RL playbook fights distributional shift with either policy constraints (stay close to the behavior policy) or value regularization (penalize out-of-distribution actions). Methods like CQL, IQL, and TD3+BC all fit under this umbrella. They still lean on global value or policy models trained over the entire dataset, and global modeling gets shaky in compositional environments where local transition structure and subtask-specific value patterns vary substantially across the state space.

The authors give a concrete example: in a MuJoCo locomotion task, the knowledge needed to sprint faster does not transfer cleanly to the knowledge needed to recover from a stumble. States that look geometrically similar after dimensionality reduction can also correspond to semantically different behaviors with distinct long-horizon returns, a property that a global value approximator fails to capture when data is limited.