Skip to main content
Back

RoleCDE Catches Every Major LLM Breaking Character Under Moral Pressure

RoleCDE exposes a systematic flaw in LLM role-playing agents: they default to their alignment training instead of staying in character when values conflict.

Paper, Full Access
  • Role Value Decoupling discovered: LLMs systematically default to alignment-safe decisions instead of role-specific values when the two conflict, even with explicit role prompts.
  • New benchmark RoleCDE: ~8,000 role profiles and ~24,000 dilemma instances across 3 difficulty levels and 8 role categories to measure this failure mode.
  • Universal across models: Qwen2.5-7B, Llama-3-8B, Gemini-2.5-flash-lite, and Kimi-K2 all show the same decoupling behavior regardless of dilemma difficulty.
  • Role category is the key variable: Decoupling severity varies significantly across character archetypes (e.g., Authority & Governance vs. Creative & Media).
  • Fine-tuning fixes it: Training on RoleCDE dilemmas improves value trade-off reasoning while preserving general role-playing fidelity and reasoning performance.
  • Open source: Benchmark and code available at github.com/rabbitrose/RoleCDE.

Ask an LLM to play a ruthless corporate negotiator, a morally grey detective, or a historical figure with views that clash with modern ethics, and something predictable happens: the model blinks. Instead of committing to the role, it quietly reverts to its safety-trained, morality-consistent self. A new benchmark called RoleCDE puts a name to this behavior, measures it at scale, and offers a fine-tuning recipe to fix it.

The tension nobody was measuring

Role-playing agents (RPAs) are LLMs steered via system prompts or fine-tuning to behave like a specific character, persona, or professional archetype. They power everything from interactive fiction and game NPCs to customer-service bots and AI companions. The problem is that alignment training, the RLHF and safety fine-tuning baked into every major model, teaches the model to be helpful, harmless, and honest as a baseline. Those values do not always match the character being played.

Prior surveys on role-playing LLMs have flagged value alignment as an open challenge, noting that current research reveals challenges in character hallucination, point-in-time role-playing, and value alignment. But existing benchmarks mostly test surface-level fidelity: does the model sound like the character? Does it know the character's backstory? What they skip is the harder question: when the character's values and the model's alignment training point in opposite directions, which one wins?

Comparison of traditional LLM evaluation vs. RoleCDE's scenario-based value conflict evaluation

The answer, consistently, is alignment wins. The paper calls this Role Value Decoupling: the agent systematically defaults to morality-consistent decisions rather than role-specific values when the two conflict, even when the role is explicitly defined in the prompt. This is not a bug in one model. It shows up across every mainstream LLM tested.

What RoleCDE actually tests

The benchmark is built around cognitive dilemma scenarios