Alberta Researchers' SEDTalker Tracks Emotions Every 20ms for Realistic 3D Faces
A new framework predicts emotion every 20 milliseconds from speech, letting 3D avatars smoothly shift affect mid-sentence instead of wearing one static expression.
PRO- SEDTalker predicts emotion at 20 ms frame resolution to drive continuous 3D facial expression changes mid-utterance.
- Hybrid Transformer-Mamba backbone extends JambaTalk with additive emotion-intensity conditioning at 457M parameters.
- Diarization corpus aggregates 58,834 utterances from 9 datasets, 149 speakers, hitting 78.9% weighted F1.
- Decoupled training removes the need for paired emotional speech and 3D face data, a longstanding bottleneck.
- Beats FaceFormer on MVE, LVE, EVE, and FFE on EmoVOCA; ablations confirm both Mamba and Transformer layers matter.
- Code, videos, and project page promised at SEDTalker.github.io; full paper on arXiv.
Most speech-driven 3D face animators can lip-sync, but ask them to convey emotion and you get one flat feeling stretched across an entire utterance. SEDTalker, from researchers at the University of Alberta and the University of Florence, tries to fix that by predicting emotion at the frame level, roughly every 20 milliseconds, and using those predictions to continuously modulate a talking head as it speaks.
Rather than relying on utterance-level or manually specified emotion labels, the method predicts temporally dense emotion categories and intensities directly from speech, enabling continuous modulation of facial expressions over time. Instead of tagging a clip as "angry" and calling it done, the system lets anger swell, drop into neutral, and slide into sadness within the same sentence.
Why static labels flatten the face
The core complaint is about temporal granularity. Real emotional speech is not a step function; it shifts, blends, and varies in intensity as a sentence unfolds. Existing systems like EmoTalk attach a single emotion vector to a whole utterance, which produces animations that either look flat or snap awkwardly between states.
A data gap sits behind that limitation. The EmoVOCA dataset offers high-quality emotional facial motion but only neutral speech audio, so models cannot directly learn how emotional acoustic cues map to expressions. That mismatch has been a persistent bottleneck for anyone trying to build genuinely emotion-aware talking heads.
Borrowing diarization for feelings
The paper's central idea borrows a trick from speaker diarization, which segments audio into "who spoke when," and applies it to emotion: segment audio into "what feeling when." A dedicated Speech Emotion Diarization (SED) model produces a timeline of emotion categories and intensity levels, and that timeline conditions the animation model.
The two halves are trained separately. The animation model learns from neutral speech paired with emotional faces (EmoVOCA), while a pre-trained SED model supplies emotion signals at inference time. That decoupling eliminates the need for aligned emotional speech-face pairs, which barely exist at scale, and makes the framework adaptable to new emotions without retraining.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
