Penn's NF-CoT Lets LLMs Think in Vectors, Beating Text Reasoning by 13%
NF-CoT embeds a normalizing flow inside an LLM to reason in continuous latent space, beating explicit chain-of-thought and diffusion-based baselines on code benchmarks while running 1.9x faster.
PRO- NF-CoT embeds a normalizing flow inside a pretrained LLM to generate continuous chain-of-thought vectors instead of text tokens, preserving exact likelihoods and KV-cache compatibility. (paper)
- On Qwen3-8B-Base across 5 code benchmarks, average pass@1 improves from 55.8 to 68.8 (+13.0%), reaching 70.1 after a lightweight RL stage.
- NF-CoT outperforms the best prior latent-reasoning baseline (LaDiR) by +7.1% while running 1.9x faster end-to-end and using 2.5x fewer FLOPs per sample.
- A single NF-CoT sample on MBPP+ achieves pass@1 = 72.1, matching the base model's pass@128 = 72.0 — equivalent to 128 base-model samples.
- Latent-space RL with GRPO improves pass@1 without collapsing solution diversity, unlike standard token-space GRPO which saturates at large sampling budgets.
- Code and model weights are coming soon; project page at nf-cot.vercel.app.
Chain-of-thought prompting made LLMs dramatically better at reasoning, but it comes with a hidden cost: every intermediate reasoning step has to be written out as text. That means the model must verbalize ideas that might be half-formed, uncertain, or purely semantic before it can move on. A new paper from researchers at the University of Pennsylvania and UC San Diego proposes a cleaner alternative: let the model think in continuous vectors instead of words, while keeping all the properties that made text-based CoT so powerful in the first place.
The result is NF-CoT, a latent reasoning framework that embeds a normalizing flow directly inside a pretrained LLM's causal stream. The key insight is deceptively simple: swap out the text tokens in the reasoning prefix for compact continuous vectors, but make sure those vectors can still be sampled probabilistically, scored with exact likelihoods, and decoded efficiently with the existing KV cache.
Why text-based reasoning has a bandwidth problem
Textual CoT forces computation through a discrete, serial, and communication-oriented token stream: each reasoning step must be verbalized before the model can proceed, even when the underlying update is semantic, uncertain, or only partially formed. The inefficiency compounds at scale: long reasoning chains are expensive, and they lock intermediate computation to surface-level language.
Latent reasoning offers a higher-bandwidth alternative by performing intermediate computation in compact continuous states before committing to text. But prior attempts at this idea each gave something up:
- Coconut-style methods (recycling hidden states as latent thoughts) are efficient but deterministic. The resulting thoughts do not define a distribution over reasoning paths.
- Diffusion-based latent methods like LaDiR are stochastic, but require iterative denoising. This separates latent-thought generation from the native autoregressive likelihood interface used by explicit CoT.
Existing latent-reasoning methods often sacrifice key advantages that make CoT effective in autoregressive language models, including native left-to-right generation, probabilistic sampling, compatibility with KV-cache decoding, and tractable likelihood estimation. NF-CoT is designed to preserve all four.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
