TeRA Matches HiRA's Accuracy on Llama-3-8B With 51x Fewer Parameters
A new parameter-efficient fine-tuning method uses tensor networks to deliver full-rank weight updates while training as few as 48 parameters per adapter.
Takeaways
- TeRA breaks the PEFT tradeoff, delivering full-rank updates with vector-method parameter counts via Tucker tensor networks.
- On Llama-3-8B commonsense, matches HiRA (r=32) accuracy at 85.31% while using 51x fewer trainable parameters.
- A 4096×4096 update can be trained with as few as 48 parameters using deep tensorization.
- Frozen random core and factor matrices are shared across layers; only tiny diagonal vectors are learned.
- Beats VeRA by 3.58 points and outperforms HiRA (r=32) on arithmetic reasoning tasks like AQuA.
- Code available under MIT on GitHub; accepted as ACL 2026 oral.
Parameter-efficient fine-tuning has been stuck in an awkward tradeoff. You can have expressive high-rank weight updates like HiRA, or the shockingly small parameter counts of vector-based methods like VeRA, but not both. A new method called TeRA (Vector-based Random Tensor Network for High-Rank Adaptation), accepted as an oral at ACL 2026, breaks that tradeoff by borrowing a trick from multi-linear algebra.
On Llama-3-8B commonsense reasoning, TeRA matches the accuracy of HiRA at rank 32 while using roughly 51x fewer trainable parameters. It beats the closest vector-based baselines by several accuracy points, and its weight updates come out near full-rank instead of the low-rank approximations that LoRA and its descendants are stuck with.
The rank vs. parameter count trap
LoRA works by freezing the pretrained weights and learning a low-rank update ΔW = AB, where the rank r controls both expressivity and parameter count. That assumption holds up for many tasks, but recent work on HiRA showed it starts to hurt on harder problems like arithmetic and multi-step reasoning where the ideal update matrix is high-rank.
The two escape routes both come with costs:
- High-rank adapters (HiRA, MoRA, QuanTA) recover expressivity by using Hadamard products with the frozen weights or square update matrices, but their parameter counts balloon with rank.
- Vector-based adapters (VeRA, LoRETTA) go the other direction, freezing randomly initialized A and B matrices and only training small scaling vectors, but they remain locked into the low-rank assumption.
TeRA’s question is simple: can you get a full-rank update using only a handful of scaling vectors?