Multi-Agent AI Systems Fail to Represent Cultural Diversity, Study Finds
A new paper shows that LLM-based multi-agent systems are far less culturally diverse than human societies, and that social interaction makes it worse
Takeaways
- New metric proposed: "Value diversity" measures how spread out values are across a whole multi-agent system, not just per agent.
- Alignment and diversity are uncorrelated: A system can have well-aligned agents and still collapse to homogeneous outputs.
- LLM systems lag far behind humans: Across 18 models and 19 cultures, all tested systems fell substantially below human societies in value diversity.
- Social interaction erodes diversity: Agents that observe each other's responses converge toward consensus, and more rounds of interaction do not recover lost diversity.
- Mixed-backbone systems help but don't close the gap: Using different model families (GPT, Claude, Qwen, etc.) for different cultural agents narrows but does not eliminate the diversity deficit.
- Open-source: Code and data are available at github.com/iNLP-Lab/MultiAgent-Diversity.
When you build a multi-agent system where each agent is supposed to represent a different culture, does the system actually behave in culturally diverse ways? A new paper from researchers at Singapore University of Technology and Design and Washington University in St. Louis says: not really, and the gap is bigger than you might expect.
Multicultural multi-agent systems are increasingly deployed in globally diverse settings, where different agents are grounded in different cultural backgrounds. Existing cultural evaluation focuses on value alignment, meaning how closely a single agent matches a target culture. But alignment is a per-agent property and cannot reveal whether a system, taken as a whole, preserves the cultural plurality it is meant to represent. The paper, "Beyond Alignment: Value Diversity as a Collective Property in Multicultural Agent Systems", argues that the field has been measuring the wrong thing.
Two metrics, one blind spot
The distinction between alignment and diversity is subtle but important. Think of it this way: you could have 19 agents, each perfectly mimicking their assigned culture in isolation, yet when you put them all together, their outputs could still cluster around a narrow band of values. That is the blind spot the paper targets.
- Alignment (existing metric): does agent X respond like a person from culture X? Measured per agent.
- Value diversity (new metric): across all agents in the system, how spread out are the values? Measured at the system level.
The authors propose value diversity as a system-level evaluation axis, defined through the dissimilarity between culturally conditioned agents' responses on a shared value survey. Concretely, each agent answers the same set of survey questions, and diversity is the aggregate pairwise dissimilarity across all those response vectors.
The benchmark: 19 cultures, 18 models
Using the World Values Survey (WVS), the team evaluates 19 cultures and 18 backbone models across a wide range of system configurations. The WVS is a long-running social science dataset that captures how people across more than 120 societies answer questions on topics ranging from happiness and religion to politics and economic values. Initiated in 1981, it has evolved into the largest non-commercial empirical time-series database encompassing over 120 societies, with its core methodology employing nationally representative surveys comprising approximately 250 items organized into 14 thematic modules.