Artificial Analysis Ranks 36 AI Models by Writing Style and Personality

Artificial Analysis has launched AA-PersonalityBench, a public evaluation of stylistic differences across 36 language models. Two systems can answer the same question correctly while producing markedly different experiences: one relies on headings and punctuation, another resembles a terse Slack reply, and a third opens with a compliment before providing useful information.

The benchmark combines 30 judged traits with 21 communication metrics computed directly from raw responses. The judged traits roll up into eight personality indexes: Expressiveness, Organization, Informativeness, Transparency, Warmth, Conviction, Imagination, and Slop. Direct metrics capture concrete features such as word count, sentence length, emoji use, tables, and punctuation.

From prompts to Elo

Thirty prompts test each judged trait, producing 900 prompts overall. Some gently encourage the target behavior, while others remain neutral. Each model answers every prompt once as a single user message without a system prompt. The result measures default, single-turn behavior; production instructions, tools, and conversation history can change that behavior. Artificial Analysis keeps the full question set private to reduce contamination and publishes a small sample, including “Why did Rome fall?” for the Structured trait.

The judge pool contains Gemini 3.8 Flash at its high setting, Claude Sonnet 5.5 at medium, and GLM 5.3 Flash at max. Each blinded comparison goes to one judge, with assignments balanced so that every evaluated model appears roughly equally across the three. Response order is randomized to reduce the tendency of judges to favor the first answer shown.

A Firth-penalized Bradley-Terry model converts the pairwise verdicts into latent trait ratings. The statistical penalty reduces bias when outcomes are sparse or heavily one-sided, and each tie counts as half a win for both systems. The resulting Elo-style scale predicts preference odds: equal ratings imply a 50% chance for either response, while a 400-point gap implies 10-to-1 odds for the higher-rated model.

The benchmark calculates 95% confidence intervals from 1,000 bootstrap resamples of the verdicts. GPT-6.1 Sol serves as the fixed anchor at Elo 1000 and receives extra comparisons to stabilize the scale.

Models at the stylistic extremes

• Expressiveness and Warmth: Claude Fable 5.1 leads, often alongside the GLM-5.3 family and Meta’s Muse models.
• Transparency: Claude Sonnet 5.5, Claude Fable 5.1, and GPT-6 Astra top the index. The benchmark reverses sycophancy when calculating this score, so less flattery improves the result.
• Imagination: Muse Spark 1.3, Claude Opus 5.5, and GLM-5.3 lead on creative, open-ended responses.
• Slop: Llama 4 Maverick, GPT-4o, and K2 Horizon post the highest Slop scores. Lower is better on this index because it measures generic, AI-sounding filler that can be removed without losing substance.

The directly computed metrics expose an equally wide spread. Qwen3.8 27B averages about 1,030 words per response, roughly 3.8 times GPT-6 Luna’s 270. Nemotron 3 Ultra produces almost three tables per answer, while most systems rarely use one. GLM-5.3 averages seven em dash characters per response; Claude Opus, Sonnet, and Haiku 5.5 use the character infrequently. K2 Horizon averages more than two emoji per answer.

House styles leave a trace

Correlating each model’s 30-trait score vector reveals clusters of stylistically similar systems. The resulting dendrogram places the GPT-6 family and Grok 4.7 in one branch; Claude, GLM, and Kimi K3 in another; and the Muse, Gemini, Qwen, and Nemotron lineages together. The groupings suggest that lab-specific house styles persist across model sizes. They describe response similarity and provide no evidence about shared training data or model ancestry.

The site’s head-to-head comparator translates Elo gaps into the probability that one model’s answer will be preferred over another on a selected trait. A separate tool lets users choose and weight any subset of the 30 traits to create a ranking for a particular application.

Putting the scores to work

Capability tests such as MMLU and coding evaluations leave interaction style largely unmeasured. A customer-support router may prioritize Warmth and low Slop. A legal drafting pipeline may favor Transparency and Conviction while penalizing emoji. A creative-writing product may place more weight on Imagination and Expressiveness. Custom weighting turns those requirements into a reproducible shortlist instead of relying on a few playground completions.

The direct metrics can also flag operational concerns before deployment, including a tendency to wrap answers in tables, end a large share of responses with follow-up questions, or overuse recognizable punctuation. These behaviors affect interface layout, token costs, brand voice, and downstream parsing even when the underlying answer is correct.

Where the benchmark stops

• One completion per prompt: The reported intervals reflect resampled judge verdicts and do not measure variation across repeated generations.
• LLM-based judging: Balanced assignments and randomized response order address known biases. Shared preferences or blind spots across the three judges can still shape the ratings.
• Private test prompts: Keeping the set private reduces contamination while limiting independent reproduction and prompt-level auditing.
• Default prompting: The scores cover single-turn responses without system instructions. Fine-tuning, application prompts, tools, and multi-turn context may produce a different style profile.

Teams evaluating a deployment should use the benchmark to narrow the field, then test finalists on representative conversations and production instructions. The full leaderboard and per-trait results are available on the evaluation page, with scoring details in the methodology. The custom ranking tool is free to use.