Skip to main content
Back

Nanjing University's TAO-RL Fixes the Broken Training Loop Behind Agentic AI

TAO-RL fixes two root causes of unstable tool-use training in LLMs: bad data and timid exploration, beating existing methods across 7 benchmarks

Paper, Full Access
  • TAO-RL is a new agentic RL framework that stabilizes tool-use training in LLMs by fixing both data quality and exploration simultaneously.
  • It filters out two types of useless training trajectories: ones where all tool calls fail, and ones where all rollouts score identically (zero learning signal).
  • An entropy bonus is applied specifically at post-tool-call tokens, pushing the model to explore diverse reasoning paths at the most critical decision points.
  • Tested on 7 benchmarks including AIME 2024/2025 and AMC 2023 across 3 model scales, TAO-RL outperforms TIR, SimpleTIR, and curriculum baselines.
  • Gradient norm analysis shows TAO-RL training is dramatically more stable than TIR, which spikes and destabilizes between steps 50-80.
  • No code or model weights released yet; the approach is compatible with standard GRPO-based training pipelines.

Training LLMs to use tools reliably is one of the messiest open problems in agentic AI. You give a model a Python interpreter or a search engine, and suddenly the training loop starts misbehaving: the model either leans on the tool for everything, or barely uses it at all. A new paper from researchers at Nanjing University proposes TAO-RL (Tool-Aware Optimization with Entropy Guidance), a framework that attacks this instability at two levels simultaneously -- the data going into training and the optimization algorithm itself.

Why tool-use RL keeps breaking

The broader field of agentic RL -- training LLMs to act as autonomous agents that call external tools -- has exploded in the last year. Systems like Search-R1 and Tool-Star have shown that reinforcement learning can teach models to interleave reasoning with tool calls in powerful ways. But the training process is notoriously fragile.

Agentic RL training remains highly unstable and prone to collapse. This instability arises from the interactive, multi-turn nature of agentic environments, which introduce compounding challenges such as invalid actions, sparse rewards, long-horizon credit assignment, and non-stationary agent-environment dynamics. Two failure modes are especially common:

  • Over-reliance on tools: The model learns to blindly invoke tools even when it shouldn't, causing the input distribution to shift away from what the base model was trained on.
  • Under-exploration: The model plays it safe, rarely trying novel reasoning paths after a tool call, and converges to mediocre but stable behavior.

The central challenge is long-horizon credit assignment -- attributing a final outcome to specific intermediate tool calls remains difficult, even with fine-grained, turn-level rewards. This difficulty leads to training instability and brittle inference-time generalization, manifesting as strategic deficiencies like tool overuse or "cognitive offloading," suboptimal personalization, and poor alignment with user preferences for tool invocation.