Nankai University's TARPO Lets AI Think in Silence, Skipping Words When Language Fails
TARPO teaches LLMs to dynamically switch between thinking in words and thinking in raw vector space, solving a core RL exploration problem in latent reasoning
Takeaways
- Core idea: TARPO lets LLMs switch between generating real tokens and passing raw hidden states forward at every decoding step, learned entirely via RL.
- Problem solved: Pure latent reasoning is deterministic, which kills exploration in RL training; TARPO borrows stochasticity from discrete token sampling instead of injecting noise.
- Architecture: A lightweight action head router reads the current hidden state and samples a binary routing decision, jointly trained with the LLM backbone via a shared group-relative advantage signal.
- Results: Consistently outperforms GRPO, DRPO, and HKPO baselines across Qwen2.5 (1.5B–7B) and Llama-3.1-8B, with more stable training dynamics and higher maintained entropy.
- No supervised warm-up needed: Unlike prior hybrid methods that require heuristic rules or teacher initialization, TARPO learns the switching policy from scratch through pure RL.
- Code available: Open-sourced at github.com/NKU-LITI/TARPO-master; paper at arxiv.org/abs/2606.05859.
Every token an LLM generates is a bottleneck. The model compresses a rich, high-dimensional internal state into a single word, then starts fresh. Chain-of-Thought (CoT) reasoning makes this bottleneck explicit: the model is forced to verbalize every intermediate step, even when the most useful "thought" has no clean natural-language equivalent. Latent reasoning