QwenGyre keeps GPUs working through hour-long agent rollouts

Hour-long coding-agent rollouts strain reinforcement learning infrastructure through uneven completion times, branching execution histories, and token counts approaching one million per task. Alibaba’s Qwen team describes QwenGyre, a framework built for this workload. The paper reports a six-percentage-point gain on a repository-generation benchmark and up to 1.85× faster end-to-end training than existing Async and Colocate baselines.

Why long rollouts starve GPUs

Conventional agentic RL systems assume that a rollout, meaning one complete model-environment execution, lasts minutes. QwenGyre targets what the authors call xLong-horizon tasks: repository generation, codebase-wide refactoring, and stack migrations that can run for hours across hundreds of tool calls.

Execution-time variance creates a long tail in which a few jobs delay the next training step after most jobs have finished. The two common scheduling models waste capacity differently:

• Colocate alternates a shared GPU pool between rollout generation and training. Training waits for the slowest rollout group before taking over the pool.
• Async reserves fixed GPU pools for generation and training. Idle capacity in one pool cannot serve the other stage.

Modern coding harnesses add a second source of overhead. Context compaction replaces older messages with summaries, sub-agents work in isolated contexts, and retries fork new execution paths. Flattening every resulting path into an independent sample duplicates shared prefixes and can lose the exact context that produced each token.

Cells and trees split the problem

QwenGyre decouples the lifetime of a harness execution from GPU assignments and training samples. Its Elastic Scheduler reallocates compute during a rollout, while its Trajectory Processor reconstructs branching histories for training.

Cells follow available work

The Elastic Scheduler divides the cluster into cells, the smallest GPU groups capable of holding the model’s training-parallel layout. A cell can serve rollout requests or host a training actor replica, allowing spare generation capacity to join training without terminating active harness executions.

Each deployment has a core cell that holds the authoritative weights and runs the optimizer. Satellite cells switch between generation and training as demand changes. Dynamic data parallelism allows a newly available satellite to join a training batch already in progress.

During a role change, a proxy redirects in-flight model requests to another inference engine. Cached attention keys and values, commonly called the KV cache, move through Remote Direct Memory Access so the destination can reuse processed prefixes instead of recomputing them.

Trees preserve exact context

The Trajectory Processor records each harness execution as a tree whose branches share stored prefixes. Every generated output remains attached to the context that produced it, including outputs created before context compaction or inside a sub-agent session.

Its loss mask includes policy-generated tokens only. Shared targets contribute once, and token losses are averaged within each execution before aggregation. This prevents executions with many branches from receiving disproportionate weight.

Timeouts can retain useful work

A timed-out repository-generation task may still leave a working partial repository with completed packages and passing tests. QwenGyre evaluates the preserved workspace and assigns partial credit whenever the completed work remains assessable, turning unfinished executions into graded training examples.

Sampling controls keep those examples within a fixed compute budget. The processor selects at most J_max trajectories per execution, with a default of five, and ranks them by harness role. For Claude Code, main-agent task-driving turns rank above sub-agent turns, while summary trajectories rank last. In the reported ablation, a cap of five matched uncapped sampling quality and reduced forward-backward time by about 25%. Training on the main trajectory alone produced lower scores.

Measured gains at scale

The authors evaluated QwenGyre on three agent workloads: NL2RepoBench for generating repositories from natural-language specifications, DeepSWE for repairing existing codebases, and TerminalBench for multi-turn terminal tasks. Every experiment used Claude Code as the harness inside containerized environments.

At the largest reported scale, Qwen 3.8 2.4T completed 48 training steps in 75.4 hours. Async required 134.6 hours, and Colocate required 91.5 hours for the same step count. Those results correspond to speedups of 1.78× and 1.21×, respectively. NL2RepoBench pass rate increased from 52.5% to 58.5% during the run.

Policy staleness measures how many optimizer updates separate the policy that generated a rollout from the policy consuming it during training. QwenGyre reported mean staleness of 1.5 updates across all three workloads.

Role changes averaged 8.52 seconds from rollout to training and 3.46 seconds in the reverse direction. Each transition remained under nine seconds, compared with executions lasting as long as two hours, although cumulative overhead depends on switching frequency.

Ablations locate the speedup

An ablation covering a 12-step workload separated the gains from streaming and elastic scheduling. Streaming alone reduced Async wall time by 11.6%. Adding standalone rollout nodes to Colocate produced little improvement in the authors’ measurements.

Adding both streaming training and cell-level role switching reduced completion time from 25.08 hours for vanilla Async to 16.12 hours for the full QwenGyre configuration. The result attributes the larger gain to reclaiming idle rollout capacity while completed trajectories continue flowing into training.

Three layers work together

1. Cell-level reallocation lets newly available GPUs join an active training batch through dynamic data parallelism.
2. Token-level provenance, called TITO in the paper, preserves the inputs and outputs associated with compaction, retries, and sub-agent delegation while treating the harness as a black box.
3. Partial-progress evaluation scores assessable work left by timed-out executions instead of assigning every timeout zero reward.

Where QwenGyre fits

Projects with hour-long, highly variable rollouts and token counts approaching one million per execution match QwenGyre’s intended operating range. Short tasks, such as math rollouts lasting about 30 seconds, provide less idle capacity to reclaim and may see limited benefit from role switching.

Cluster and sampling constraints

• Minimum cluster size: Each switchable cell must hold the model and its training-parallel configuration. Small clusters may lack enough independent cells to reallocate capacity effectively.
• Limited global shuffling: Streaming assigns completed rollout groups to successive updates. Training recipes that require globally randomized minibatches must use single-step bursts, meaning one optimizer update per scheduling burst. The paper reports smaller speedups in this mode, though it still outperformed the baselines.

Reference code is unavailable

The paper and Qwen’s announcement identify no accompanying implementation. The Qwen on Hugging Face page lists the research but no QwenGyre repository.

An engineering team adopting the design must integrate cell-aware scheduling, request rerouting, fast KV-cache transfer, token-provenance tracking, and task-specific partial-credit evaluators. The framework’s strongest fit is a large cluster running expensive agent jobs where rollout variance leaves substantial GPU capacity idle.