Dream-RSI Rehearses Search Policies on Past Agent Runs

Researchers affiliated with Google, Google DeepMind, the University of Maryland, College Park, and the University of Virginia have introduced Dream-RSI, a framework that improves how an AI discovery agent explores candidate solutions. It keeps the underlying model fixed and replays previous search trees to test thousands of alternative exploration policies before deploying one in a new run. The paper reports competitive or better results on algorithm design, mathematical optimization, and GPU kernel engineering while reducing expensive agent calls by as much as two orders of magnitude.

The approach separates search improvement from model training. Dream-RSI treats the history of attempted solutions, evaluation scores, workspace states, and costs as a deterministic replay environment. The authors call this process “dreaming”: a policy-development agent rehearses new search strategies against recorded outcomes, then promotes the strongest policy to the next online discovery round.

Expensive Feedback Stalls Search

Recursive self-improvement systems commonly repeat a simple loop: propose a candidate, evaluate it, use the result to guide the next proposal, and continue. Hard discovery tasks can require thousands of cycles, so ineffective branch selection consumes model calls, evaluator runs, wall-clock time, and accelerator capacity.

Hand-written exploration policies remain fixed as evidence accumulates. Online policy optimization can adapt, but its feedback arrives only after a long discovery rollout, and evaluating several weak policies requires several costly runs. Dream-RSI shifts most of that policy testing into replay, where recorded evaluations can be reused.

A Search Tree Becomes a Replay World

Each completed discovery run produces a tree whose nodes contain attempted code, evaluation results, workspace snapshots, and execution costs. A replayed policy can select among recorded nodes, change their order, vary batch sizes, and stop at a different point. Because the outcomes already exist, the simulator can score those decisions without executing the candidate programs again.

The simulator still consumes compute for policy generation and tree replay. Its “zero-cost” property refers specifically to avoided coding-agent calls and avoided candidate evaluations, which are usually the expensive parts of the loop.