Skip to main content
Back

PoLoRA Beats Adam on LoRA Finetuning With 1.7x Fewer Steps

A new LoRA optimizer replaces Adam with three matrix-aware ingredients, hitting the same loss in 1.2 to 1.7 times fewer steps.

  • PoLoRA is a new optimizer for LoRA finetuning that beats Adam by 1.2 to 1.7x in steps.
  • Wall-clock overhead per step is capped at 3% across four base models from 1B to 8B parameters.
  • Combines product-aware spectral updates, Kronecker-factored curvature preconditioning, and magnitude control on both factors.
  • Optimal learning rate stays stable across LoRA ranks 32, 64, 128, 256, while Adam's drifts.
  • Speedup is larger on distribution-shifted data like Bengali finetuning than on well-represented code data.
  • Open-source implementation available at github.com/nikhilgsh/polora.

A trio of researchers just proposed a new optimizer built specifically for LoRA finetuning that consistently beats the default Adam recipe, something several recent Muon-style variants have struggled to do. Nikhil Ghosh, Tetiana Parshakova, and Robert M. Gower introduce PoLoRA, a Preconditioned Orthogonalized LoRA optimizer that reaches Adam's held-out loss in 1.2 to 1.7 times fewer steps while adding at most 3% per-step overhead, across models from 1B to 8B parameters.

Why Adam is a lazy choice for LoRA

LoRA finetunes a frozen base weight by adding a trainable low-rank update parameterized as the product of two matrices, B and A. The standard recipe applies Adam to these two factors as if they were unrelated parameter vectors, treating them as a single flat vector and ignoring both the matrix and product structure of LoRA.

What actually matters to the layer is the product BA, not the individual factors. When you take a small step, the linearized change in the merged weight is B·ΔA + ΔB·A, meaning an update to B depends on A, and vice versa. Naive attempts to swap in a matrix-aware optimizer like Muon fail to help. The authors report that applying Muon to each factor does not consistently improve over Adam, and neither do the product-aware Muon variants proposed in concurrent work.