Skip to main content
Back

Tokyo Researchers Beat LoRA Fine-Tuning With a Simple Linear Regression on Frozen VLMs

Researchers show simple linear regression on frozen VLM hidden states beats LoRA fine-tuning for predicting individual aesthetic preferences on photos and artworks.

Pro

  • VLMs encode diverse aesthetic attributes in hidden states, propagating into language decoder layers (paper)
  • Simple ridge regression on frozen layer-15 hidden states beats LoRA fine-tuning and few-shot prompting
  • Linear-Hidden hits 0.611 Spearman on PARA photos and 0.568 on LAPIS artworks with 100-shot support
  • Cross-domain transfer crushes dedicated PIAA-ICI baseline: 0.568 vs 0.206 on LAPIS
  • Vision-only DINOv3 lags VLMs, showing text supervision helps organize aesthetic concepts
  • Full code and experiments released at github.com/ynklab/vlm-latent-piaa

Predicting which image a specific person will find beautiful sounds like a job for heavy machinery: fine-tune a big vision-language model on that user's ratings, or engineer a specialized network with meta-learning tricks. A new paper from the University of Tokyo argues you can skip almost all of that. The authors show that modern VLMs already contain rich, multi-level aesthetic information in their hidden states, and that a plain linear regression on top of those frozen activations outperforms LoRA fine-tuning and few-shot prompting for personalized image aesthetics assessment (PIAA).

The paper, "What Do Vision-Language Models Encode for Personalized Image Aesthetics Assessment?" by Koki Ryu and Hitomi Yanaka, tackles a task where models predict the aesthetic score a specific user would assign to an image, reflecting personal taste rather than an average rating. The full pipeline is open-sourced at github.com/ynklab/vlm-latent-piaa.

Why existing PIAA pipelines struggle

Traditional PIAA methods lean on multi-stage recipes: pretrain on large-scale generic aesthetics datasets, then adapt to each target user, often with meta-learning or attribute-interaction modules. Those pipelines are expensive and, critically, tend to overfit to one image domain. A model trained on photographs rarely transfers cleanly to artworks. VLMs are an attractive alternative because they see huge amounts of text and images during pretraining, but until now evaluations of their aesthetic perception were limited to multiple-choice or binary formats, and no prior work had used them for individual-level PIAA.

Diagram of the Linear-Hidden PIAA pipeline showing an image passing through a VLM, hidden representations extracted, and user-specific linear estimators producing personalized scores

Probing what the model already knows