HoPA Distills Video, Audio, and Text Into 100 Training Samples
HoPA compresses datasets with three or more modalities into tiny synthetic sets, using a single SVD-derived proxy instead of expensive pairwise alignments.
ProTakeaways
- HoPA is the first dataset distillation method targeting three or more modalities jointly, not just image-text pairs.
- It uses SVD on the per-instance Gram matrix and keeps a rank-1 proxy instead of enumerating all modality pairs.
- A spectral theorem shows single-mode objectives yield a tighter trajectory-matching bound than pairwise InfoNCE or BCE.
- On MSR-VTT, VGGSound-S, and DiDeMo, HoPA beats LoRS, RepBlend, MTT-VL, and TESLA with roughly 2x speedup.
- 500 distilled triplets (5-6% of data) preserve over 95% of full-dataset retrieval recall across all benchmarks.
- Code will be released; full paper and pseudo-code available on arXiv.
Dataset distillation has been quietly turning training pipelines upside down. Instead of hauling around millions of samples, you compress the useful signal into a few hundred synthetic examples and train on those. Almost every method in the literature has assumed you are dealing with images, or at most images plus text. A new paper introduces HoPA (High-order Proxy Alignment), which pushes distillation into the omnimodal regime, meaning three or more modalities distilled jointly in a single pass.
Why bimodal tricks break when a third modality shows up
Extending distillation beyond two modalities remains underexplored because heterogeneity and cross-modal interactions both compound. The obvious approach is to reuse bimodal recipes: enumerate every pair of modalities (video-text, video-audio, audio-text) and match them one by one. This scales quadratically in the number of modalities and breaks apart the joint structure that ties everything together for a single instance.
The authors formalize a subtle issue. Trajectory-matching distillation works by making a student model, trained on the synthetic data, end up at roughly the same weights as a teacher trained on the real data. They prove that the gap between those endpoints is bounded by an accumulated sum of modality-wise gradient mismatches. Every additional modality adds another term to that bound, so naive extensions get monotonically worse as you add modalities under a fixed data budget.
A rank-1 shortcut through the Gram matrix
HoPA's central move is to stop modeling modalities pair by pair and instead treat all of them as one object. For each instance, the k modality embeddings are stacked into a matrix z, and its Gram matrix G = zzᵀ captures every pairwise similarity at once. The team then applies singular value decomposition and keeps only the leading component.