Stanford's PersuasionTrace Catches AI Persuasion Mid-Conversation and Exposes Flawed Simulators
PersuasionTrace tracks human belief changes turn-by-turn during LLM conversations, revealing that vanilla LLM simulators fail to replicate how real people update beliefs under persuasion
- PersuasionTrace is a new open-source framework that tracks human belief changes turn-by-turn during multi-turn LLM persuasion conversations, not just before and after.
- LLMs (GPT-5) successfully persuaded humans across text, audio, and personalized topic conditions, all significantly outperforming a control group.
- Human belief trajectories cluster into two types: ~half barely move, while the other half shift early then partially drift back; emotional (pathos) appeals predict which cluster someone falls into.
- Ethos (authority appeals) was the only rhetorical dimension with a statistically significant effect, and it was negative, meaning appeals to authority backfire.
- A new Bayesian Network simulator scores 81.3 on human-likeness vs. 64.7 for vanilla LLM simulators, showing plain LLMs fail to replicate real human belief dynamics.
- Code, data, and an interactive demo are available at the PersuasionTrace GitHub repo.
We already know LLMs can persuade people. What we don't know is how it happens. A new paper from Stanford and the University of Washington introduces PersuasionTrace, a framework that moves beyond the standard before-and-after belief measurement to capture the full trajectory of belief change during a multi-turn conversation with an AI persuader. The findings reveal that human belief dynamics are far messier and more heterogeneous than existing research suggests, and that the simulated human targets commonly used to test persuasion systems are deeply flawed.
The problem with "did it work?"
Almost every LLM persuasion study to date measures a single number: how much did someone's belief shift from before the conversation to after? These endpoint measures identify whether persuasion occurred, yet miss where and how beliefs moved within a dialogue. In agentic LLM settings, where policies act over many steps, endpoint-only metrics can obscure whether a system is robust across turns or simply benefits from a few brittle moments of movement. The field has also relied heavily on simulated human targets, LLM agents prompted to play the role of a persuadable person. Given their flexibility, LLMs promise not only to persuade real people, but also to simulate human targets of persuasion, to model the mechanisms of belief change over a conversation. The problem, as this paper shows, is that they're bad at it.
The stakes are real. AI advice can shift beliefs durably even after the advisor is removed, and models with minimal demographic data are markedly more persuasive than humans in multi-round debates. Large-scale user experiments have confirmed that LLM-generated messages can shift beliefs to a degree comparable with, or even exceeding, human-written arguments. Understanding the mechanism, not just the outcome, matters enormously for both AI safety and the design of persuasive systems.
What PersuasionTrace actually does
PersuasionTrace records both standard pre/post and turn-level belief reports during persuasive dialogues, implemented in a web-based platform used to analyze how LLM persuaders and human targets behave across turns. After each message from the LLM persuader, participants rate their belief in the proposition on a 0-100 scale, generating a full trajectory rather than a single delta. The platform also supports audio conversations and personalized propositions, where participants bring their own real-life decisions to debate.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
