Skip to main content
Back

MAny Fixes the Hidden Memory Flaw Quietly Crippling Multimodal AI Models

A new training-free framework fixes a blind spot in multimodal continual learning by merging both vision projectors and LoRA weights.

  • MAny exposes a dual-forgetting problem: projector drift and LoRA reasoning collapse both hurt multimodal continual tuning.
  • Cross-modal Projection Merging keeps per-task projectors and blends them at inference using visual prototype similarity.
  • Low-rank Parameter Merging uses recursive least squares for a closed-form, exemplar-free LoRA fusion rule.
  • Beats HiDE by 8.57 FAA points on UCIT with LLaVA-1.5-7B, cutting forgetting from 5.58% to 0.37%.
  • Achieves negative forgetting on MLLM-DCL, meaning old-task accuracy improves as new tasks arrive.
  • Training-free, CPU-based merging; baselines live in MCITlib.

Most defenses against catastrophic forgetting in multimodal large language models have focused on protecting the language backbone, quietly assuming the vision-to-language projector is safe. A new paper called MAny (Merge Anything) argues that assumption has been hiding half the problem, and proposes a training-free merging framework that fixes both halves at once.

Two forgetting problems, not one

When a multimodal LLM is fine-tuned on a sequence of instruction datasets, it tends to forget earlier tasks. The standard fix is to constrain or route around updates in the LoRA adapters attached to the language model. The authors run a decoupling analysis on the UCIT benchmark and find that even when you swap the drifted final-task LoRA back for the original task-specific one, performance still fails to reach the oracle level. On some tasks like CLEVR-Math, restoring only the projector actually helps more than restoring the LoRA.

The authors call this the dual-forgetting phenomenon: perception drift in the cross-modal projection space (the MLP that maps CLIP features into the LLM's token space), and reasoning collapse in the low-rank parameter space of the LoRA adapters. Cosine similarity of projector outputs for early tasks drops to around 0.83 after sequential fine-tuning, meaning the model literally starts misinterpreting visual inputs it used to understand.

Correlation heatmaps and 3D surface plots showing perception drift and reasoning collapse

Splitting the fix along the failure modes

MAny addresses each failure mode with a dedicated merging strategy. The whole thing runs as CPU-based algebra after each task finishes training, with no extra gradient steps and no replay buffer.