Artificial Analysis' Optima Lets Any Team Build Custom AI Benchmarks

Artificial Analysis, the independent benchmarking firm best known for its continuously updated LLM leaderboards and evaluations like AA-Briefcase and GDPval-AA, has launched Optima , a platform that lets anyone build a custom benchmark tailored to their own tasks, data, and use cases. The pitch is simple: standardized public benchmarks tell you which model is generally capable, but they cannot tell you which model is right for your finance agent, legal assistant, or image classification pipeline.

The timing is not accidental. LLM benchmarks in 2026 are necessary but insufficient. MMLU has saturated above 90%. HumanEval suffers from training data contamination. SWE-Bench scores vary 25 percentage points depending on scaffolding. No single benchmark predicts production performance reliably. Optima is Artificial Analysis' answer to that gap.

Three ways in, one leaderboard out

Optima offers three distinct entry points for building a benchmark, designed to meet teams wherever their data already lives:

• Upload a dataset , bring an existing evaluation set from your own files or from Hugging Face directly.
• Import agent traces , pull recorded agent sessions from platforms like Arize, Braintrust, or Langfuse. The LLM observability market is estimated at $2.69B in 2026 , and these are the tools where production traces already live for most teams.
• Describe your use case , give Optima a plain-text description plus a few example inputs and outputs, and a build agent drafts the tasks and rubrics for you.

There is also an IDE integration: install the Optima skill and it can pull context directly from your coding environment and previous sessions to scaffold a benchmark without leaving your editor.

The grading engine is the real differentiator

Building the task set is only half the problem. Grading at scale is where most custom eval efforts fall apart. Optima brings two grading modes that mirror Artificial Analysis' own research-grade evaluations:

• Rubric grading , you define objective criteria and a judge model scores each response against them deterministically. Priced at $0.25 per criterion per model.
• Pairwise judging , the same head-to-head comparison approach used in GDPval-AA and AA-Briefcase. GDPval-AA v2 compares model submissions head-to-head on the same task. For each matchup, the two outputs are anonymized and an LLM judge picks a winner. These blind pairwise results are aggregated into an Elo rating per model. Optima brings this same mechanism to your private data. Priced at $0.75 per match.

The result is a leaderboard that tracks not just quality scores but Cost per Task and Time per Task for every model in the run. That three-way comparison , quality, cost, speed , is the core value proposition. The platform claims it can surface alternatives that deliver equivalent quality at 10x lower cost or 10x faster.

What Artificial Analysis is actually packaging here

To understand why this matters, it helps to know what Artificial Analysis has built over the past two years. Artificial Analysis is an independent AI benchmarking and analysis company that publishes continuously updated evaluations of language models, coding agents, image, video, and speech models. The platform covers over 575 models and tracks performance across intelligence, speed, cost, and specialized capabilities.

Their flagship evaluations are not trivial. AA-Briefcase is a frontier agentic evaluation for long-horizon knowledge work, testing agents on realistic business workflows that require deliverables such as spreadsheets, presentations, and memos. GDPval is a benchmark designed to evaluate AI models on real-world, economically valuable tasks across 44 occupations. The dataset encompasses 1,320 tasks derived from nine major industries contributing significantly to the U.S. GDP. Optima essentially opens the grading infrastructure behind these evaluations to external teams.

The site's FAQ explicitly states that providers cannot pay for results or methodology changes. That independence is the brand. Optima extends it: your benchmark runs on the same neutral infrastructure, with the same pairwise judging panel, applied to your private data.

Who wins from this

The clearest winners are teams already spending engineering time building and maintaining internal evals. MIT Project NANDA's State of AI in Business 2025 found that only 5% of integrated enterprise AI pilots produce measurable P&L impact, with workflow integration and evaluation gaps cited as the binding constraints. The teams shipping the other 95% are mostly not failing on model quality. They are failing on whether anyone can tell, in advance, whether the LLM system will work on real world tasks in production.

Beta testers reportedly used Optima to answer questions like: which model can save 10x the cost without a meaningful quality drop for a finance and accounting agent? Which model best matches the writing style of lawyers for a legal agent? Which model best identifies elements in a custom image dataset? These are exactly the questions that public leaderboards cannot answer.

The integration with Arize, Braintrust, and Langfuse is strategically smart. 89% of surveyed organizations use agent observability , meaning most teams already have traces sitting in one of these platforms. Optima turns that existing instrumentation into benchmark fuel without requiring teams to re-instrument or export data manually.

The competitive picture

Optima enters a crowded but fragmented space. Tools like Braintrust, Confident AI, and Weights and Biases Weave all offer LLM evaluation capabilities. Braintrust consolidates tracing, evaluation, prompt management, and dataset versioning in one system. But none of these are benchmarking-first companies , they are observability or experiment-tracking platforms that added evaluation. Artificial Analysis is coming from the opposite direction: a benchmarking company adding a self-serve product layer on top of its existing evaluation infrastructure.

The key differentiator is the pairwise judging panel. Running head-to-head comparisons with the same judge models used in published research-grade evaluations is something no observability tool currently offers. A 100-200 example custom evaluation suite, built once with domain expert input, pays back across every future model selection decision. Optima is betting it can be the platform where that suite gets built and maintained.

Pricing and availability

Optima is available today at artificialanalysis.ai/optima. Pricing is usage-based with no markup on raw token costs for running models. Grading is the incremental cost: $0.25 per rubric criterion per model, and $0.75 per pairwise match. There is no subscription fee listed , you pay for what you run.

The platform also supports bringing your own agent into a benchmark run over HTTP, so your custom agent stack can compete head-to-head against frontier models on the same tasks under the same conditions. For teams trying to justify a specific model or architecture choice internally, that kind of apples-to-apples comparison is hard to replicate with any other tool currently available.