IBM's Abstract-CoT Teaches AI to Reason With 11x Fewer Tokens
IBM Research's Abstract Chain-of-Thought replaces verbose reasoning traces with a compact sequence of learned tokens, cutting inference costs by up to 11.6x while matching verbalized CoT performance.
- Abstract-CoT replaces verbose natural language reasoning traces with short sequences of reserved, non-human-readable tokens, achieving up to 11.6x fewer reasoning tokens at comparable performance.
- The training recipe combines a policy iteration warm-up (bottlenecked SFT + self-distillation) with warm-started GRPO reinforcement learning; neither stage alone is sufficient.
- On Qwen3-8B, Abstract-CoT reaches 90.8% on MATH-500 and 60.8 win-rate on AlpacaEval, matching or beating verbal CoT SFT+RL while using a fraction of the tokens.
- Results generalize across Qwen3-4B, Qwen3-8B, Qwen3-32B, and Granite 4.0 Micro (3B), confirming the method transfers across model families and sizes.
- An emergent Zipf's law distribution over the abstract vocabulary appears during RL training, suggesting the model learns structured, reusable token patterns analogous to natural language.
- No code or weights released yet; training used public AI2 datasets and the full paper is on arXiv.
Every time a modern reasoning model thinks through a hard problem, it generates hundreds or thousands of tokens of internal monologue before producing an answer. That verbosity has a real cost: slower responses, higher compute bills, and longer training rollouts during reinforcement learning. A new paper from IBM Research proposes a different approach entirely. Instead of thinking in English, what if the model thought in a compact, learned language of its own?
The paper, "Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought", introduces Abstract Chain-of-Thought (Abstract-CoT), a post-training mechanism that replaces natural language reasoning traces with short sequences of reserved, non-human-readable tokens. The model still produces a final answer in plain language, but the scratchpad it uses to get there is a compact sequence of abstract symbols like <TOKEN_A>, <TOKEN_F>, <TOKEN_M>, and so on.
The problem with thinking out loud
Large language models increasingly rely on long, explicit chains-of-thought to solve complex, multi-step reasoning problems. Despite its effectiveness, verbalized CoT is an expensive mechanism, increasing latency and cost at inference while bloating the length of traces during reinforcement learning. There is also a deeper issue: prior works suggest that verbalized CoT can be unfaithful, while leveraging a different latent reasoning process that is not communicated.
The field has explored alternatives. Continuous latent reasoning methods, like Coconut, replace CoT tokens with continuous hidden-state vectors. By internalizing multi-step inference in a model's latent space, latent CoT aims to overcome the expressive and computational bottlenecks associated with token-level chain-of-thought output. But continuous methods come with their own problems: they are harder to control, difficult to analyze, and their performance tends to lag behind verbalized CoT. Abstract-CoT sits in the middle ground: fully discrete (like text), but non-verbal (like latent reasoning).
A new vocabulary for thinking
The core idea is simple. The model produces a short sequence of tokens from a reserved abstract vocabulary in lieu of a natural language CoT, before generating a response. These tokens, named things like <TOKEN_A> through <TOKEN_Z> and beyond, are added to the model's tokenizer as entirely new entries with randomly initialized embeddings. At inference time, the model generates a bounded sequence of these abstract tokens inside
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
