Skip to main content
Back

Chung-Ang University's HyCal Beats Continual Learners on Frozen CLIP Without Training

A training-free calibration method fuses cosine and Mahalanobis distances on frozen CLIP embeddings to fix prototype drift across imbalanced, heterogeneous visual domains.

  • HyCal is a training-free prototype method that fuses cosine similarity and Mahalanobis distance on frozen CLIP embeddings.
  • The paper names Domain Gravity: overrepresented domains warp embedding geometry and drag prototypes off target.
  • New XD-VSCIL benchmark chains 8 disciplines from aircraft to medical imaging with realistic imbalance.
  • New CDE metric weights domains by 1/sqrt(K) so dominant domains cannot inflate the score.
  • Beats RanPAC, KLDA, FeCAM, and trainable Primal-RAIL by 4.5+ points under high-scale imbalance.
  • Adding 10 classes costs 5.4 GFLOPs versus 12 for retraining, with 301 MB storage at 300 classes.

Continual learning research has spent years quietly assuming that new tasks look a lot like old ones. A new paper from Chung-Ang University and collaborators pushes back on that assumption with a benchmark and a method aimed at what actually happens when a vision-language model has to absorb wildly different disciplines in sequence, with wildly uneven amounts of data. The method, called HyCal, is training-free, runs on frozen CLIP, and beats existing few-shot continual learners on a new benchmark called XD-VSCIL.

The bias hiding in your embedding space

Few-Shot Class-Incremental Learning (FSCIL) is the setting where a model learns new classes from a handful of examples per class without forgetting the old ones. Most FSCIL frameworks presuppose homogeneous visual domains and balanced few-shot distributions, yet real data arrives from heterogeneous sources such as medical imagery, textures, aerial views, or natural scenes, which differ markedly in visual statistics and representational complexity.

The authors put a name on the pathology this creates: Domain Gravity. Each domain induces a representational potential based on its visual coherence and data density, and overrepresented or low-entropy domains exert disproportionately strong influence in the shared embedding space. Feed a CLIP-based continual learner 470 texture samples and only 110 medical images, and the texture domain will warp the geometry of the embedding space, pulling prototypes for the medical classes toward regions where they do not belong.

Under heterogeneous domain priors that differ in scale, variance, and orientation, metric calibration becomes biased and covariance-based regularization becomes unreliable in few-shot regimes. That is why standard prototype methods, even ones using clever random projections, collapse when imbalance gets severe.