Skip to main content
Back

Mistral and Llama3 Lack the Brain Structure That Keeps Humans Unbiased

New research finds humans regulate bias through structurally distinct memory layers, while Mistral and Llama3 collapse those layers into overlapping associations.

Pro

Paper, Full Access
  • Researchers modeled System 1 and System 2 thinking as multilayer semantic networks for humans, Mistral 7B, and Llama 3.1 8B.
  • Humans show three structurally distinct memory layers; in the LLMs, associative and categorical layers collapse into one.
  • Bias drops systematically from associative to structured layers in humans, but varies unpredictably across topics in LLMs.
  • LLMs often produce associative responses when asked for categorical relations, contaminating taxonomic knowledge.
  • Findings use the LLM World of Words dataset and WordNet-style prompted generations for definitions and relations.
  • Read the full preprint on arXiv for methodology and effect size plots.

A new paper from researchers at the University of Pisa, the Italian National Research Council, and the University of Trento takes the popular System 1 versus System 2 metaphor and turns it into something you can actually measure. The team built multilayer semantic networks for humans, Mistral 7B, and Llama 3.1 8B, then asked whether the structure of memory itself has anything to do with implicit gender bias. For the two open models tested, the mechanism humans appear to use to keep bias in check is largely missing.

Why implicit bias keeps slipping past guardrails

Dual process theories propose that biases arise primarily from associative System 1 thinking, while deliberative System 2 thinking mitigates them. The cognitive mechanisms behind that mitigation remain poorly understood, in humans and in LLMs alike. The practical version of the problem is familiar to anyone shipping a chatbot: a model can ace an explicit fairness benchmark and still produce stereotyped associations the moment you probe indirectly. Recent work by Bai and colleagues at PNAS showed exactly that pattern, with explicitly unbiased models still forming biased associations under indirect probes.

The authors wanted to know whether the human trick of using more deliberative, structured knowledge to dampen bias has any counterpart inside a transformer. That required representing both kinds of thinking as something concrete, rather than as vibes about fast versus slow.

Turning dual-process theory into graphs

Each model (and humans) gets a three-layer semantic network built from the same vocabulary of roughly 12,000 cue words: