Skip to main content
Back

HoPA Distills Video, Audio, and Text Into 100 Training Samples

HoPA compresses datasets with three or more modalities into tiny synthetic sets, using a single SVD-derived proxy instead of expensive pairwise alignments.

Pro

  • HoPA is the first dataset distillation method targeting three or more modalities jointly, not just image-text pairs.
  • It uses SVD on the per-instance Gram matrix and keeps a rank-1 proxy instead of enumerating all modality pairs.
  • A spectral theorem shows single-mode objectives yield a tighter trajectory-matching bound than pairwise InfoNCE or BCE.
  • On MSR-VTT, VGGSound-S, and DiDeMo, HoPA beats LoRS, RepBlend, MTT-VL, and TESLA with roughly 2x speedup.
  • 500 distilled triplets (5-6% of data) preserve over 95% of full-dataset retrieval recall across all benchmarks.
  • Code will be released; full paper and pseudo-code available on arXiv.

Dataset distillation has been quietly turning training pipelines upside down. Instead of hauling around millions of samples, you compress the useful signal into a few hundred synthetic examples and train on those. Almost every method in the literature has assumed you are dealing with images, or at most images plus text. A new paper introduces HoPA (High-order Proxy Alignment), which pushes distillation into the omnimodal regime, meaning three or more modalities distilled jointly in a single pass.

Why bimodal tricks break when a third modality shows up

Extending distillation beyond two modalities remains underexplored because heterogeneity and cross-modal interactions both compound. The obvious approach is to reuse bimodal recipes: enumerate every pair of modalities (video-text, video-audio, audio-text) and match them one by one. This scales quadratically in the number of modalities and breaks apart the joint structure that ties everything together for a single instance.

The authors formalize a subtle issue. Trajectory-matching distillation works by making a student model, trained on the synthetic data, end up at roughly the same weights as a teacher trained on the real data. They prove that the gap between those endpoints is bounded by an accumulated sum of modality-wise gradient mismatches. Every additional modality adds another term to that bound, so naive extensions get monotonically worse as you add modalities under a fixed data budget.

A rank-1 shortcut through the Gram matrix

HoPA's central move is to stop modeling modalities pair by pair and instead treat all of them as one object. For each instance, the k modality embeddings are stacked into a matrix z, and its Gram matrix G = zzᵀ captures every pairwise similarity at once. The team then applies singular value decomposition and keeps only the leading component.