SafeConstellations Cuts LLaMA's Unnecessary AI Refusals by 73% Without Retraining
A new inference-time steering method reduces LLM over-refusals by up to 73% by tracking task-specific representation trajectories through model layers.
ProTakeaways
- SafeConstellations reduces LLM over-refusals by up to 73% via inference-time representation steering, no retraining required.
- Each NLP task produces a consistent trajectory through transformer layers, forming distinct "constellation" patterns.
- Method identifies task from hidden states, then steers only benign tasks toward non-refusal manifolds on 4 layers.
- LLaMA-3.1-8B over-refusal dropped from 17.77% to 4.81% with zero MMLU accuracy loss.
- Adds only ~0.2s latency per sample; uses 847MB for 5-task embedding store on LLaMA-3.1-8B.
- Code and 1,047-sample benchmark released publicly.
Safety-aligned LLMs have a nagging habit: they sometimes refuse perfectly benign requests because the input text happens to mention something sensitive. Ask a model to translate a historical document about wartime atrocities, or run sentiment analysis on a review that uses phrases like "killer good," and you may get a polite refusal instead of a useful response. A new paper from researchers at Macquarie University and IOE Pulchowk Campus proposes SafeConstellations, a technique that peers into a model's internal representations to fix this without retraining.
The refusal problem hiding in plain sight
The authors formalize a specific failure mode they call task over-refusal: the rejection of benign instructions based solely on input context, ignoring that the actual task has benign intent. Consider a customer support pipeline running sentiment analysis on angry reviews, or a translation service handling historical documents. The wrapper task is harmless, but the sensitive content inside trips the safety machinery anyway.
The problem shows up most in production applications where contexts arrive from external sources, user inputs, or RAG pipelines. To measure it systematically, the authors evaluated four model families across a purpose-built benchmark. LLaMA exhibits the highest over-refusal on benign tasks like translation and sentiment. Claude stays consistently cautious but rarely over-refuses. GPT-4o shows mild over-refusal concentrated in low-resource translation, particularly for languages like Hindi, Urdu, and Nepali.

Constellations in the embedding space
The core discovery is geometric. When the researchers traced how token representations move through the transformer's layers, they saw something striking. LLMs follow distinct "constellation" patterns in embedding space as representations traverse layers, with each task maintaining consistent trajectories that shift predictably between refusal and non-refusal cases.