CUHK's NoRA Fixes LoRA's Hidden Gradient Gap With one Line
A one-line change to LoRA's initialization closes the gradient gap with full fine-tuning, boosting speed and stability with zero extra parameters.
PRO- NoRA normalizes LoRA's down-projection columns to unit norm, closing the gradient gap with full fine-tuning.
- The trick adds zero trainable parameters and zero inference cost since adapters still merge into
W. - Gains reported across pretraining, SFT, and RL, including reduced catastrophic forgetting per the paper.
- A one-time init variant, NoRA-init, captures most of the benefit without touching the training loop.
- Block-identity initialization (BIMI) matches NoRA-init, suggesting column-norm uniformity is the real active ingredient.
- Code and details at the project page spherelab.ai/NoRA.
Low-rank adaptation is the default way most teams squeeze new behaviors out of large models without paying to update every weight. But despite years of variants, one basic question has stayed open: how should you regularize LoRA's training so it converges quickly and stays stable? A new paper from researchers at CUHK, Microsoft Research, and Yuanshi Intelligence, Normalized Low-Rank Adaptation, argues the fix is almost embarrassingly simple: normalize the columns of the down-projection matrix.
The gradient gap nobody was fixing
Standard LoRA replaces a weight update with two small matrices, A (down-projection) and B (up-projection), so the effective change is ΔW = αBA. B is initialized to zero so the adapter starts as a no-op, and A gets a random Gaussian or Kaiming init.
Because B = 0 at step zero, the gradient flowing into A is also zero on the first update, which means the entire early trajectory of the adapter is dictated by A. Working through the math, the first weight change looks like ΔW = -η · G · (α²AᵀA), where G is the full fine-tuning gradient. LoRA is implicitly applying a preconditioner P = α²AᵀA to the true gradient, and the squared column norms of A act as per-input-direction learning rates.
With random init, those column norms are both uneven and small, so LoRA's early gradient norm runs far below what full fine-tuning would produce. That is the learnability gap the paper targets.
The fix in one line
NoRA normalizes each column of
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
