Johns Hopkins' ProSDD Catches Emotional Voice Deepfakes Where Others Fail
A two-stage training recipe teaches deepfake detectors to model natural prosody first, cutting error rates by more than half on emotional spoofing attacks.
PRO- ProSDD trains XLS-R to predict speaker-conditioned pitch, energy, and voice activity from real speech before spoof fine-tuning
- Cuts ASVspoof 2024 EER from 39.62% to 7.38% when trained on the same set
- Achieves roughly 50% relative EER reductions on EmoFake and EmoSpoof-TTS emotional benchmarks
- Uses a deliberately lightweight classifier head to prove gains come from the enriched backbone
- Two-pass training: masked prosodic prediction plus unmasked spoof classification per step
- Code and details at the ProSDD project site
Speech deepfake detectors have a dirty little secret: they crush the benchmarks they were trained on, then fall apart the moment a synthesized voice starts to sound emotional. A new paper from Johns Hopkins and Hong Kong Polytechnic, ProSDD, argues the fix is to stop obsessing over spoof artifacts and start teaching the model what real speech actually sounds like.
Why current detectors keep getting fooled
Speech deepfake detection systems perform well on standard benchmarks but often fail to generalize against expressive and emotional spoofing attacks. Many methods rely on spoof-heavy training data, learning dataset-specific artifacts rather than transferable cues of natural speech. That works fine until a TTS system with laughs, whispers, or emotional inflection shows up, at which point the classifier has no idea what to do.
The authors flag two structural problems with the standard recipe. Fine-tuning a self-supervised backbone with only a spoof classification loss biases it toward whatever quirks live in the training set. And modern TTS and voice conversion systems still leak subtle prosodic inconsistencies, yet no one is training detectors to notice them. Humans do the opposite: they internalize what natural speech sounds like and flag anything that deviates.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
