humans&'s Persimmon Fools AI Detectors at Human Rates With 550B Parameters

A new class of large language model just landed, and it isn't trying to be a helpful assistant. Persimmon, from the startup humans&, is a 550-billion parameter model whose sole job is to convincingly pretend to be a person, or several people, in a chat. The research preview targets a gap that most frontier labs have quietly ignored: assistant models are terrible at imitating actual humans.

The pitch is straightforward. When you use frontier models to simulate users in evaluations or training environments, the outputs vary much less than those of real people and represent only a sliver of the wider human distribution. That narrowness quietly poisons any downstream system built on top of those simulations, from agent evaluations to synthetic conversation data.

The Turing test, rebuilt for group chats

To measure whether a user model actually matches human behavior, the team introduces a Multi-User Turing Test. An LLM judge is shown a real human group conversation and a model-generated one, and has to guess which is which. The score is the fraction of times the judge assigns the wrong label. Perfect performance is 50 percent (pure chance), and every frontier assistant tested comes in under 3 percent.

Persimmon sits at roughly 20 percent fooled rate, which is short of human parity but one to two orders of magnitude above the competition. Reported scores on the three test datasets:

• TIDES: Persimmon 21.1 percent, Nemotron Ultra Base 2.7 percent, GPT 5.6 at 0.11 percent, GPT 6 (Astra) at 0 percent, Opus at 0.11 percent.
• Internal Workspace Conversations: Persimmon 18.6 percent, with every comparison model under 1 percent.
• TutorMoments: Persimmon 19.8 percent, next best GPT 6 (Astra) at 0.39 percent.

A separate check using Pangram, an AI-text detector held out from training, tells the same story. Pangram labeled 1.3 percent of human transcripts, 2.3 percent of Persimmon transcripts, 54.3 percent of GPT 6 (Astra) transcripts, and 94.3 percent of Fable 5 transcripts as AI. Persimmon slips past the detector at roughly the human base rate.

Why simulate users at all

User models are the missing half of the agent evaluation stack. To test a customer-support agent, train a negotiation policy, or roll out a hypothetical product decision across a population, you need something on the other end of the conversation that behaves like a real person, complete with hesitation, oversharing, backtracking, and change over time. Assistant models fill that role today, and they fill it badly.

The blog post frames the goal as exploring how situations might play out for individuals and groups by feeding the model a scenario description and profiles for the simulated users, then watching how different decisions and environments affect people over time. Concrete use cases include synthetic evaluation data for agents, red-teaming multi-user interactions, and studying how AI decisions ripple across different personalities.

Where it wins, and where it still loses

Beyond the Turing test, humans& evaluates two harder properties: how information is revealed over time, and whether personality holds up over long conversations.

The Trickle Test measures whether a model reveals facts at the same pace a real human did in a reference transcript. Assistant-optimized models tend to have higher recall than precision, since their instinct to be thorough and helpful degrades into oversharing when they play the user. Persimmon hits the highest precision of 43 models tested (88.5 percent) while keeping competitive recall (77 percent), meaning it dribbles information out more like a real user rather than dumping the whole order ID, account ID, and callback request in turn one.

Long-context coherence is where Persimmon still struggles. On 80-turn conversations, humans stay coherent 87.3 percent of the time, Persimmon holds at 60.7 percent, GPT 6 (Astra) sits at 98.4 percent, and Opus 5 at 99 percent. The wrinkle is that the assistant models are too coherent. Real people drift, contradict themselves, and forget details. Persimmon drifts in a human-like shape, just too aggressively.

How it was built

The recipe is a two-stage build on top of an open base model:

1. Initialize from NVIDIA's 550-billion parameter Nemotron 3 Ultra base and midtrain on a diverse collection of conversations between people from the public internet.
2. Posttrain with reinforcement learning against an adaptive discriminator, using the model's own generations as training data.

That second stage is the interesting part. Straight next-token prediction on human dialogue produces a model that drifts as errors compound. RL against a discriminator, essentially a GAN-flavored objective applied at the conversation level, was added specifically to fix long-horizon coherence. The team observed that this improved long-horizon behavior without noticeably changing general distribution matching. Training ran on thousands of Blackwell-generation GPUs on humans&'s recently built cluster.

The safety wrinkle nobody can ignore

A model designed to be indistinguishable from a real person is exactly the kind of thing that makes trust and safety teams nervous. humans& is upfront about it. The guardrails typically used for assistant models are incompatible with broad human modeling, and distribution is being limited to avoid misuse for impersonation, privacy violations, exploitative manipulation, or consequential decisions about real people.

Access is gated. A playground lets you define profiles and a scenario, and an API samples the next simulated user turn given profiles and a scenario. Both require approval, with pre-sampled conversations available while you wait. Academic applicants can request API credit grants, and humans& has said it may reach out about intended use cases before approving.

What this changes

The industry has been treating "simulated user" as a prompt engineering problem, something you solve by writing "you are a frustrated customer" at the top of a GPT context. The Persimmon numbers suggest that approach was leaving most of the human distribution on the table, and that evaluations, RL environments, and synthetic datasets built on prompted assistants have a systematic realism gap that no one was measuring.

If user models become a real category, several assumptions need updating. Agent benchmarks that use LLM-simulated users are probably too easy in some dimensions and too hard in others. Synthetic conversation datasets used for post-training may be reinforcing the very homogeneity they were meant to fix. The interesting frontier for RL environments may turn out to be more realistic humans on the other side of the conversation rather than harder tasks.