ULPT Cuts LLM Fine-Tuning Parameters by 98% Without Losing Performance
Researchers show prompt embeddings can be learned in just 2 dimensions, then randomly up-projected, cutting trainable parameters by 98% with no loss in accuracy.
ProTakeaways
- ULPT learns prompt embeddings in as few as 2 dimensions, up-projected by a frozen random matrix.
- Cuts trainable parameters by 98% versus vanilla prompt tuning while matching accuracy on 20+ tasks.
- Only a random seed needs to be stored to reconstruct the up-projection at load time.
- Small learnable shift and scale vectors align the randomly projected embeddings with the model's input space.
- Beats LoRA, VeRA, and FourierFT on GSM8K and MBPP with orders of magnitude fewer parameters.
- Code released at github.com/MANGA-UOFA/ULPT for drop-in use with T5 and Llama models.
Prompt tuning has always had an awkward tension baked into it. You want a tiny, parameter-efficient way to steer a frozen large language model, but the learned prompt embeddings have to live in the same hidden dimension as the model itself. Push from 768 dimensions to 4096 and your storage inflates right along with it, even though the underlying task hasn't gotten any harder.
A new paper by Zijun Wu, Yongchang Hao, and Lili Mou proposes an almost embarrassingly simple fix. Their method, Ultra-Low-Dimensional Prompt Tuning (ULPT), learns prompt embeddings in a space as small as two dimensions and then uses a frozen random matrix to project them back up to the model's hidden size. ULPT achieves a 98% reduction in training parameters compared to vanilla prompt tuning while preserving performance across more than 20 NLP tasks.
Why prompt embeddings got so bloated
Standard prompt tuning, introduced by Lester et al. in 2021, prepends a handful of learnable embedding vectors to the input and optimizes them by gradient descent while the LLM stays frozen. The catch: those vectors are typically constrained to match the model's hidden dimensionality, so a 100-token prompt on a 4096-dim model means 400K learnable parameters per task, even though prior work has shown downstream tasks have very low intrinsic dimensionality.
Earlier work like DPT tried decomposing the prompt matrix E into two learnable matrices Z (low-rank prompts) and P (an up-projection). The up-projection itself still scales with the model's hidden size, and once you're maintaining a separate P per task, the storage savings evaporate.