---
title: "TeRA Matches HiRA's Accuracy on Llama-3-8B With 51x Fewer Parameters"
subtitle: "A new parameter-efficient fine-tuning method uses tensor networks to deliver full-rank weight updates while training as few as 48 parameters per adapter."
url: "https://alphasignal.ai/news/tera-matches-hira-s-accuracy-on-llama-3-8b-with-51x-fewer-parameters"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-14T00:00:00.000Z"
updated: "2026-08-23T22:56:30.088Z"
type: "Paper"
topics:
  - "Post Training"
  - "Llms"
subtopics:
  - "Lora"
  - "Fine Tuning"
  - "Small Models"
upvotes: 1070
source_url: "https://arxiv.org/abs/2509.03234"
comments: 0
truncated: true
---

# TeRA Matches HiRA's Accuracy on Llama-3-8B With 51x Fewer Parameters

> A new parameter-efficient fine-tuning method uses tensor networks to deliver full-rank weight updates while training as few as 48 parameters per adapter.

Parameter-efficient fine-tuning has been stuck in an awkward tradeoff. You can have expressive high-rank weight updates like HiRA, or the shockingly small parameter counts of vector-based methods like VeRA, but not both. A new method called [TeRA](https://arxiv.org/abs/2509.03234) (Vector-based Random Tensor Network for High-Rank Adaptation), accepted as an oral at ACL 2026, breaks that tradeoff by borrowing a trick from multi-linear algebra.

On Llama-3-8B commonsense reasoning, TeRA matches the accuracy of HiRA at rank 32 while using roughly 51x fewer trainable parameters. It beats the closest vector-based baselines by several accuracy points, and its weight updates come out near full-rank instead of the low-rank approximations that LoRA and its descendants are stuck with.

## The rank vs. parameter count trap

LoRA works by freezing the pretrained weights and learning a low-rank update `ΔW = AB`, where the rank `r` controls both expressivity and parameter count. That assumption holds up for many tasks, but [recent work on HiRA](https://arxiv.org/abs/2501.13787) showed it starts to hurt on harder problems like arithmetic and multi-step reasoning where the ideal update matrix is high-rank.

The two escape routes both come with costs:

* **High-rank adapters** (HiRA, MoRA, QuanTA) recover expressivity by using Hadamard products with the frozen weights or square update matrices, but their parameter counts balloon with rank.
* **Vector-based adapters** (VeRA, LoRETTA) go the other direction, freezing randomly initialized A and B matrices and only training small scaling vectors, but they remain locked into the low-rank assumption.

TeRA’s question is simple: can you get a full-rank update using only a handful of scaling vectors?

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2509.03234)
- [TeRA official GitHub repository](https://github.com/guyuxuan9/TeRA)
- [HiRA: Parameter-efficient Hadamard High-Rank Adaptation](https://arxiv.org/abs/2501.13787)
- [VeRA: Vector-based Random Matrix Adaptation](https://arxiv.org/abs/2310.11454)