Rehearsals' Engram Beats GPT and Claude at Predicting Human Survey Behavior

A new Kaggle benchmark measures how closely AI systems can predict the answer distribution of a defined group of people. Human Behavior Bench, created by Rehearsals with Kaggle and Google DeepMind, evaluates GPT, Gemini, Claude, Grok, and Rehearsals’ Engram against survey responses from US adults.

The results matter for developers using language models as synthetic research participants, audience simulators, or substitutes for survey panels. The evaluated frontier models perform reasonably well on broad populations, then lose substantial accuracy on demographic and behavioral segments.

Engram leads a divided leaderboard

Rehearsals reports an estimated human noise ceiling near 93, reflecting variability in sampled responses. The frontier models sit roughly 12 to 17 points below that estimate. Engram’s 94.1 reflects its fit to aggregate survey distributions and should not be interpreted as evidence that it predicts individual people better than humans do.

Inside the 1,199-task test

Each task pairs a closed-ended question with a target cohort and an observed answer distribution. Cohorts use demographic attributes such as age, income, and geography, or behavioral attributes such as GLP-1 use and recent car purchases. Public tasks include their questions, audiences, and answers; the stratified private holdout limits direct tuning against the evaluation set.

The scoring metric is one minus total variation distance, scaled to 100:

Score = 100 × (1 - 0.5 × Σ|p_i - q_i|)

In the formula, p_i is the model’s predicted share for an answer option and q_i is the observed survey share. Identical distributions score 100, while distributions with no overlap score 0. Evaluating the full distribution captures errors that top-answer accuracy would miss.

Kaggle’s harness gives each base model a question, answer options, and cohort description, then requests probabilities across the options. Engram follows a different procedure: it simulates a population of AI twins, collects their individual answers, and aggregates those responses into a distribution.

Specific cohorts pull models off target

Rehearsals reports scores around 85 for general-population questions and roughly 55 to 60 for specific cohorts. The models frequently move their predictions toward the average US response, muting differences associated with age, location, purchasing history, and other segment attributes.

Representative misses

• Rural insurance customers: Models predict that about half of rural drivers renew car insurance without shopping around. The observed share is about one quarter.
• Older Midwest shoppers: Every evaluated LLM predicts that no more than 6% of adults aged 60 and older pay at grocery stores by phone or watch. The survey share is 25%.
• Northeastern streaming customers: Models predict that half or fewer of adults aged 55 to 64 would accept a $4 monthly price increase. The observed share is 70%.

Three recurring errors

1. Flattened intensity: Models identify the leading answer about 40% of the time on product features, messaging, and pricing, compared with about 70% on brand perception and daily routines. They underpredict the strongest response by an average of 13 percentage points.
2. Excessive apathy: Predictions overshoot responses containing “never,” “none,” or “don’t” by about 10 percentage points.
3. Distorted age effects: For cohorts under 60, models capture about three quarters of the observed difference from comparison audiences. For cohorts aged 60 and older, they exaggerate that difference.

The six LLMs also produce distributions about twice as close to one another as they are to the observed responses. All six select the wrong leading answer on roughly one quarter of tasks, indicating correlated errors across vendors and model families.

Population fidelity needs its own evaluation

Human Behavior Bench evaluates population fidelity: whether a system can reproduce the response distribution of a specified cohort. Reasoning benchmarks, coding tests, and general knowledge exams provide little evidence about that capability. The results also document a tendency toward average-seeming responses without identifying whether pretraining, preference optimization, safety tuning, prompting, or another component causes it.

Teams applying LLMs to pricing research, ad testing, product surveys, or synthetic panels need segment-level validation against recent human data. A strong aggregate score can conceal large errors for the audiences driving a business decision.

Practical safeguards

• Evaluate complete answer distributions, including calibration and subgroup error, instead of relying only on the most likely answer.
• Keep representative human samples in the workflow for each target segment.
• Report sample sizes, uncertainty intervals, and performance by cohort.
• Revalidate predictions when prices, products, geography, or audience definitions change.
• Avoid treating group-level survey estimates as predictions about an individual’s behavior.

The leaderboard has limits

The benchmark covers group-level responses from US adults. It does not evaluate predictions for individuals, consistency across one person’s answers, or performance in other countries. Its tasks concern stated answers to surveys and interviews, so matching them does not establish an ability to forecast purchases or other future actions.

Score differences may also reflect sampling variation. With a median of 176 respondents per task, closely grouped results such as 79.0, 79.1, 79.3, and 79.5 require uncertainty estimates before they can support firm model rankings.

The leaderboard uses different execution paths for its leading system and the base models. Kaggle runs the base models through a common prompting harness, while Rehearsals runs Engram through its in-house simulation system. Rehearsals also created the benchmark and entered Engram, so independent reproduction would strengthen comparisons involving the leader.

Public submissions come later

The public tasks and official leaderboard are available on Kaggle, while public model submissions are planned for a later release. According to the published roadmap, Kaggle will maintain the benchmark and add models. Rehearsals plans to expand the panel to 3,500 adults in late October or early November 2026, followed by an international version covering Canada, the UK, France, Switzerland, Italy, Japan, Korea, Taiwan, and Singapore between December 2026 and January 2027.

Developers can inspect the Rehearsals analysis and public Kaggle tasks to audit subgroup calibration. Until open submissions arrive, the leaderboard figures remain reported results produced through the benchmark partners’ evaluation pipelines.