Skip to main content
Back

Canada's NRC Reveals Why Fine-Tuning Safety Benchmarks Are Lying to You

A new paper reveals that fine-tuning safety evaluations are fundamentally broken — and that capability must be measured alongside safety to draw any valid conclusions.

Paper, Full Access
  • Fine-tuning LLMs for task performance degrades safety alignment, even on benign data — but the paper argues current safety measurements are themselves unreliable.
  • The fix: anchor safety evaluation to a specific capability target (e.g., a benchmark score), not arbitrary training epochs or dataset sizes.
  • Fine-tuned models can produce incoherent, garbled outputs on safety prompts — which automated LLM judges incorrectly classify as safe refusals.
  • Safety benchmark choice (BeaverTails vs. SorryBench vs. XSTest) can flip your conclusions about whether a mitigation method works.
  • Perplexity is proposed as a diagnostic tool to detect broken outputs that fool safety evaluators.
  • No code released; the contribution is a methodological framework for capability-grounded safety evaluation applicable to any fine-tuning pipeline.

Fine-tuning a large language model on your own data is one of the most common things teams do to ship a production AI product. You take a capable, safety-aligned base model, train it on your domain-specific data, and expect to get a smarter, more useful assistant. What you might not expect is that you've also quietly dismantled its safety guardrails , and that the benchmarks you're using to check for that might not even be telling you the truth.

A new paper, "Safety Measurements for Fine-tuned LLMs Should be Grounded in Capability", from researchers at the National Research Council Canada, makes a pointed argument: the entire way the field currently evaluates safety after fine-tuning is methodologically unsound. The core claim is that you can't measure safety in isolation , you have to anchor it to a capability goal first.

The problem nobody's been naming

Recent work has established the concerning phenomenon that fine-tuning a large language model for a specific task can significantly impact the safety guardrails on the base model , even in cases where the fine-tuning dataset does not contain any harmful content. This is the known problem. What this paper tackles is the less-discussed meta-problem: even when researchers try to measure how much safety has degraded, the measurements themselves are unreliable.

How do we compare the safety of two models in a meaningful way? Given the stochastic nature of LLMs, are safety measurements repeatable? Are they reproducible across minor inconsequential variations in the fine-tuning process, like different random seeds? Are the conclusions similar whether the fine-tuning proceeds for one epoch, or two, or more? These are the questions the paper forces into focus.

The authors argue that previous studies made essentially arbitrary choices about how much to fine-tune, on what data, and for how long , without tying those choices to any concrete capability target. That makes it nearly impossible to compare results across studies or to know whether a proposed safety mitigation actually works.