EBMC Forces AI to Stop Ignoring Tone and Facial Cues in Emotion Recognition
A new framework fixes the persistent problem where text overshadows audio and video signals when models try to read human emotion from mixed inputs.
ProTakeaways
- EBMC introduces a two-stage framework to fix modality imbalance in multimodal sentiment analysis, detailed here.
- An energy-based coordination module rebalances gradients automatically instead of using hand-tuned weights.
- Achieves 50.34% Acc-7 on CMU-MOSI and 57.32% on CMU-MOSEI, beating prior SOTA.
- Cuts text dominance from over 50% to a more balanced split across text, audio, and video.
- Under missing-modality conditions on MOSEI, averages 78.92% F1 versus 74.50% for the closest baseline.
- Code and 6.4M-parameter offline variant available at github.com/kangverse/EBMC.
Emotion recognition systems that combine text, audio, and video have a dirty secret: the words almost always win. Even when a speaker's tone drips with sarcasm or their face betrays discomfort, the model latches onto the transcript and ignores the rest. A new paper introduces EBMC, an Enhance-then-Balance Modality Collaboration framework that first strengthens weaker modalities and then forces them to have a real seat at the table during training.
The imbalance is well documented. Text consistently dominates prediction, while audio and visual signals stay under-exploited because their emotional cues are weaker or noisier. Dominant modalities accumulate larger gradients and reinforce their own representations, while weaker ones receive insufficient updates. The authors call this a Matthew effect, where quieter signals become progressively marginalized.
Two stages, four modules
EBMC splits the pipeline into an enhancement stage and a balancing stage, each with two components.

Stage one targets representation quality:
- Modality Semantic Disentanglement (MSD) splits each modality into a shared component (aligned across text, audio, and video via an InfoNCE contrastive loss) and a modality-specific component (prosody, facial cues) kept distinct through a cosine-similarity penalty.