Skip to main content
Back

Nankai University's TARPO Lets AI Think in Silence, Skipping Words When Language Fails

TARPO teaches LLMs to dynamically switch between thinking in words and thinking in raw vector space, solving a core RL exploration problem in latent reasoning

  • Core idea: TARPO lets LLMs switch between generating real tokens and passing raw hidden states forward at every decoding step, learned entirely via RL.
  • Problem solved: Pure latent reasoning is deterministic, which kills exploration in RL training; TARPO borrows stochasticity from discrete token sampling instead of injecting noise.
  • Architecture: A lightweight action head router reads the current hidden state and samples a binary routing decision, jointly trained with the LLM backbone via a shared group-relative advantage signal.
  • Results: Consistently outperforms GRPO, DRPO, and HKPO baselines across Qwen2.5 (1.5B–7B) and Llama-3.1-8B, with more stable training dynamics and higher maintained entropy.
  • No supervised warm-up needed: Unlike prior hybrid methods that require heuristic rules or teacher initialization, TARPO learns the switching policy from scratch through pure RL.
  • Code available: Open-sourced at github.com/NKU-LITI/TARPO-master; paper at arxiv.org/abs/2606.05859.

Every token an LLM generates is a bottleneck. The model compresses a rich, high-dimensional internal state into a single word, then starts fresh. Chain-of-Thought (CoT) reasoning makes this bottleneck explicit: the model is forced to verbalize every intermediate step, even when the most useful "thought" has no clean natural-language equivalent. Latent reasoning