Skip to main content
Back

humans& Persimmon Ultra Fools AI Judges 48% of the Time Simulating Real Users

humans& opens public API access to Persimmon v0.1 Pro and Ultra, using extra generation compute to push user simulation to near-human realism.

  • humans& launched Persimmon v0.1 Pro and Ultra, two test-time-scaled tiers of its user-simulation model, publicly accessible.
  • Ultra scores 48% on the 8-turn Multi-User Turing Test versus 18% for standard Persimmon, where 50% is human parity.
  • Pro and Ultra generate more tokens per turn, trading compute for realism and better profile adherence.
  • Long-context coherence over 80 turns now tracks the human curve instead of staying artificially consistent like frontier models.
  • Built on NVIDIA Nemotron 3 Ultra 550B with a Mamba-2 plus MoE hybrid, 1M-token context, available via API and SDK.
  • New safeguard framework built from scratch since standard assistant guardrails do not fit user models; launch will be restrictive.

humans& has released two higher-compute tiers of its Persimmon user-simulation model. Persimmon v0.1 Pro and Ultra are available without a waitlist through a playground and API. In the company’s headline benchmark, Ultra raised the standard model’s score on an eight-turn Multi-User Turing Test from 18% to 48%, close to the 50% chance rate.

Built to play the user

Persimmon generates the user side of a conversation. Developers provide a profile, conversation history, and situational context; the model returns responses intended to remain consistent with that person across long, multi-party exchanges. The main use cases include testing support bots, evaluating agents, generating synthetic research sessions, and creating reinforcement-learning environments without repeatedly recruiting human participants.

The model is fine-tuned from NVIDIA’s Nemotron 3 Ultra, which has 550 billion total parameters and activates 55 billion for each token. Its hybrid LatentMoE architecture combines Mamba-2 for efficient sequence processing, a mixture-of-experts system that routes work through selected model components, and attention for tracking relationships within the context. It supports context windows of up to one million tokens.

For v0.1, humans& first continued training on public multi-person conversations. The team then used reinforcement learning with an adaptive discriminator that detects implausible or out-of-character responses, addressing a common failure mode in long simulated conversations.

More compute raises the fooling rate

Pro and Ultra apply test-time scaling, allocating more generation-time compute and tokens to each response. Ultra uses the largest budget, followed by Pro and standard Persimmon. The release does not disclose the exact token budgets, latency differences, or per-tier operating costs.

Reported Persimmon v0.1 tier differences
Tier Inference budget Eight-turn fooling rate
Standard Baseline 18%
Pro Higher Not disclosed
Ultra Highest 48%

The added compute targets three weaknesses that often expose synthetic users:

  • Multi-user realism: Ultra’s 48% result approaches random guessing by the benchmark judge. Humans& reports that frontier assistant models prompted to role-play users scored below 1% on comparable tests.
  • Long-conversation behavior: Pro and Ultra track the human consistency curve more closely over 80-turn conversations. Real people occasionally contradict themselves or shift emphasis, while assistant models often remain implausibly consistent.
  • Profile adherence: The higher-compute tiers more reliably express traits supplied in a written persona instead of converging on a generic, highly cooperative respondent.

What the 48% score measures

The Multi-User Turing Test presents an LLM judge with one continuation written by a person and another generated by a model. The judge must identify the human continuation. With balanced pairs, random guessing produces a 50% fooling rate; Ultra’s 48% means the judge separated the two only slightly better than chance across eight-turn rollouts.

Humans& also reports the following eight-turn rates for standard Persimmon across three datasets:

Reported standard Persimmon results at k = 8
Dataset Judge fooled rate
TIDES 21.1%
Internal Workspace Conversations 18.6%
TutorMoments 19.8%

The release does not explain how those dataset-level figures relate to the 18% top-line baseline, or provide dataset-level Ultra results, confidence intervals, judge agreement, or independent replication. Those omissions limit conclusions about statistical significance and performance outside the tested domains.

Better users can sharpen agent evaluations

Assistant models make weak user substitutes because they tend to provide complete information, write polished responses, cooperate readily, and preserve context with unusual precision. An agent tested against those behaviors may perform well in simulation and then fail when people answer briefly, omit details, change direction, or introduce conflicting goals.

A more realistic simulator can produce harder evaluation cases at scale, including multi-party conversations and long-running tasks. It can also help researchers compare agent versions against repeatable user profiles. Synthetic results still require validation against held-out human conversations, especially in specialized domains where the supplied benchmarks may offer little coverage.

Humans& already uses Persimmon to train its collaborator model, Ampersand. Persimmon supplies user behavior during reinforcement-learning rollouts that can span hundreds of thousands of tokens, while Ampersand’s actions alter the subsequent conversation. Improvements to the simulator can therefore change both the difficulty and realism of that training environment.

Open access comes with stricter guardrails

Pro and Ultra are available through the playground and API. The service provides a Python SDK and REST interface that accept profiles, scenario context, and conversation history, then return the next simulated user turn.

Human-like generation also supports abuse cases such as impersonation, scams, and fabricated reviews. Humans& says it screens messages in context, applies session-level safety checks, and built safeguards specifically for user simulation. Conventional assistant policies transfer poorly because a simulator may need to represent manipulation, hostility, or other unsafe human behavior inside legitimate tests.

The company plans to use restrictive defaults at launch and adjust them as it reviews real applications. Teams evaluating the service should measure tier-specific latency and cost, test safety filters against their scenarios, inspect outputs for stereotyping or persona leakage, and compare synthetic behavior with representative human data before using it in production decisions.