Fudan's EBM-RL Forces AI Characters to See a Scene Before Speaking
EBM-RL teaches vision-language models to see a scene, think about it, then speak in character — unlocking immersive NPC dialogue for VR and interactive storytelling
- EBM-RL is a new RL framework that teaches vision-language models to perceive a video scene, reason about it, then generate in-character dialogue — all as separate, rewarded steps.
- It uses four rewards: CLIP-based scene-text alignment, perceptual-cognitive gain, answer faithfulness, and format consistency — designed for open-ended creative output, not just verifiable tasks.
- On the immersive role-playing benchmark, EBM-RL beats larger VLMs, with +24.8 and +24.2 gains in visual grounding and situational persona consistency over the Crab baseline.
- It achieves zero-shot transfer to out-of-domain VideoQA benchmarks (NExT-QA, PororoQA, ActivityNet-QA) with no additional fine-tuning.
- The team releases the first open-source video-grounded role-playing dataset, built from 37 films across 13 franchises using script-grounded and LLM-augmented pipelines.
- Current limitation: experiments are offline only — real-time interactive VR deployment remains future work, and character naturalness gains are modest.
Ask a text-only role-playing model to voice a soldier pinned down in a burning building, and it will likely deliver a generic, tone-deaf line that ignores the chaos around it. The character's words might be stylistically correct, but the atmosphere , the smoke, the urgency, the dread , is nowhere in the response. A new paper from researchers at Fudan University and collaborators tackles exactly this gap with EBM-RL (Eye-Brain-Mouth Reinforcement Learning), a framework that forces a vision-language model to see a video scene, think about what it means for the character, and only then speak.
The gap between character and context
Text-based role-playing models can imitate character styles, yet they often fail to reflect a scene's atmosphere and evolving tension , both essential for immersive applications such as VR games and interactive narratives. This is a structural problem. Most role-playing systems are purely text-driven: they receive a character description and a dialogue history, then generate the next line. They have no way to perceive what is visually happening in the scene, so they cannot adapt tone, urgency, or emotional register to match it.
While effective for objective video understanding and question answering, existing video models typically behave as neutral narrators and lack character-specific tone and personality. So the problem is a two-sided void: video models can describe a scene but cannot inhabit a character, while role-playing models can inhabit a character but cannot perceive a scene.
See, Think, Speak , in that order
EBM-RL is a decoupled GRPO-based framework that explicitly separates observation (<perception>), reasoning (<think>), and utterance (<answer>). The name maps directly to anatomy: Eye (perception), Brain (reasoning), Mouth (response). By forcing the model to produce each stage in a structured, tagged format, the training process can reward each stage independently rather than treating the whole response as a single opaque output.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
