Skip to main content
Back

Dual-Branch Graph AI Reads Emotion From Faces, Voice, and Words at Once

A new framework splits shared and private multimodal signals, then routes them through Fourier and hypergraph branches to push conversational emotion recognition past strong baselines.

Pro

Paper, Full Access
  • New MERC framework splits modality-invariant and modality-specific features before graph learning, cutting cross-modal redundancy.
  • A Fourier graph branch extracts low and high frequency emotional patterns from shared features.
  • A speaker-aware hypergraph with Jacobi-polynomial spectral filters models higher-order multi-speaker interactions.
  • Hits 70.81 WF1 on IEMOCAP and 65.70 WF1 on MELD, beating graph and transformer baselines.
  • Ablations show the shared Fourier branch contributes the largest single performance gain in the design.
  • No public code or checkpoints referenced; reproduction requires implementing dual hypergraph transformation and four disentanglement losses.

Recognizing emotion inside a multi-party conversation is deceptively hard. The words say one thing, the tone says another, the face says a third, and each speaker's mood drifts across turns. A new paper, Disentangled Dual-Branch Graph Learning, attacks this messy problem by first pulling apart the signals that modalities share from the ones they keep to themselves, then feeding each stream into a graph network tuned for its job.

On IEMOCAP, the proposed model reaches a weighted F1 of 70.81, beating every compared baseline, and it also tops every reported category on the harder MELD benchmark. The interesting part is how it gets there.

Why conversational emotion recognition keeps stalling

Multimodal Emotion Recognition in Conversations (MERC) tries to assign an emotion label to each utterance using text, audio, and video jointly. The inputs are heterogeneous and often carry redundant or overlapping emotional information, which introduces noise and blunts discriminative learning. Signals across modalities are not always temporally or semantically aligned, and emotions in a dialogue depend on complex multi-speaker interactions where pairwise modeling cannot capture higher-order dependencies.

Most prior systems throw all three modalities into a shared embedding space and then apply attention or a graph neural network. That mixes redundant cues with unique cues, and it treats speaker-to-speaker influence as a series of pairwise edges even when three or four people are talking over each other.

Splitting the signal into shared and private

The first move is a dual-space disentanglement stage. Every modality passes through two encoders: a