One Banned Word Causes GPT-4o-mini to Lose 31% of Its Knowledge
Instruction-tuned LLMs lose 14 to 48 percent of their comprehensiveness when told to avoid a single common word or punctuation mark.
PRO- Simple lexical constraints cause instruction-tuned LLMs to lose 14 to 48 percent of comprehensiveness, per the paper
- Effect holds across Llama, Qwen, Mistral, OLMo, and GPT-4o-mini (31 percent loss)
- Two-pass generation (draft then rewrite) recovers 59 to 96 percent of response length
- Linear probes predict collapse from prompt representations with R-squared up to 0.94
- Base models show no systematic collapse, isolating instruction tuning as the cause
- Independent LLM-as-judge scoring misses 80+ percent of the quality loss pairwise evaluation catches
A new diagnostic study suggests the polished helpfulness of instruction-tuned language models is more brittle than anyone assumed. When researchers appended a single trivial instruction to a prompt, something like Do not use any commas in your response, models across every family they tested abandoned their thorough, structured answers and defaulted to a much shorter, less informative reply. The effect held for Llama, Qwen, Mistral, OLMo, and GPT-4o-mini, and it survived scaling from 7B to 70B parameters.
The paper, One Token Away from Collapse, frames this as a planning failure inside the model rather than a genuine capability gap. Crucially, it does not appear in the base (pre-instruction-tuning) checkpoints, which points the finger squarely at alignment training.
The bridge that fails under a feather
The setup is deliberately minimalist. The authors took 40 prompts across four categories (explanations, how-to, comparisons, and technical questions) and appended eight lexical constraints one at a time: bans on commas, colons, semicolons, bullet points, the word the, discourse markers like however and therefore, and two compositional combinations. The output was still free-form natural language, no JSON or XML in sight.
The results are stark. All three instruct models show substantial comprehensiveness loss (14 to 40 percent overall), with the unconstrained baseline winning 77 to 98 percent of pairwise comparisons. Qwen-2.5-7B-Instruct was the most fragile at minus 39.9 percent, followed by Llama-3.1-8B at minus 23.4 percent and Mistral-7B at minus 14.2 percent.
Fragility gets worse with scale, not better. Llama-3.3-70B collapses more than Llama-3.1-8B (minus 34.7 percent vs. minus 25.9 percent). Proprietary recipes are not immune: GPT-4o-mini collapses at minus 31.0 percent with a 99 percent win rate for the baseline, contrary to prior findings that closed-weight models resist the format tax on structured output. Fully open training does not help either, with OLMo collapsing at minus 37.7 percent.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
