Tianjin University's KnowRL Beats Math Reasoning Limits at 1.5B With Smarter Hints
A new reinforcement learning framework replaces long solution prefixes with tiny, curated knowledge point hints, hitting a new 1.5B state of the art.
Takeaways
- KnowRL replaces long solution prefixes with minimal subsets of atomic knowledge points during RL training.
- KnowRL-Nemotron-1.5B hits 70.08 average without hints, 74.16 with selected KPs on eight benchmarks.
- Zero-correct rollout fraction drops from 41.21% to 13.00%, effectively solving reward sparsity on hard problems.
- The paper identifies a pruning interaction paradox where removing multiple bad hints jointly hurts performance.
- Constrained Subset Search (CSS) uses ~38% fewer KPs than full annotation while beating all baselines.
- Code, model, and KP annotations are open source at github.com/Hasuer/KnowRL.
Reinforcement learning with verifiable rewards has become the default trick for pushing reasoning models past their supervised training ceiling, but it hits a wall on genuinely hard problems: if every rollout is wrong, there is no learning signal. A team from Tianjin University, Baidu, and the Chinese Academy of Sciences argues that the popular fix, injecting hints into the prompt, has been done wrong. Their new framework, KnowRL, treats hints as a minimal-sufficient guidance problem rather than a quantity problem, and the results at the 1.5B scale are hard to ignore.
Why longer hints backfire
Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, then scale guidance by piling on more tokens, which introduces redundancy, inconsistency, and extra training overhead. The KnowRL authors pinpoint three failure modes of the status quo.
- The critical-segment effect: performance improves sharply once a short key hint segment appears, with diminishing returns beyond it.
- Cross-hint inconsistency: longer prefixes may introduce branching or ambiguity.
- Training-efficiency trade-off: abstraction-based hints rely on teacher models, increasing computational overhead.
Once a model sees the one crucial idea it needed, extra guidance mostly adds noise and can send policy updates in the wrong direction. That reframing motivates a very different pipeline.
Atomic knowledge points instead of prefixes
KnowRL decomposes guidance into what the authors call knowledge points, or KPs: short, general mathematical principles required to solve a problem, stripped of any problem-specific numbers or leakage. The curation pipeline runs offline before RL training. For each problem, DeepSeek-R1 is sampled until it produces at least one verified correct solution, then prompted a second time to extract the indispensable principles used, and finally used again as a reviewer that flags any KP leaking the specific answer or numbers.