Prime Intellect Turns Painful LLM Benchmarking Into a Single CLI Flag
Prime Intellect launches Hosted Evaluations, turning complex agentic benchmark infrastructure into a two-click workflow on their Environments Hub
- Hosted Evaluations launched: Prime Intellect now runs benchmarks on managed infrastructure, removing the need to set up harnesses, sandboxes, or compute locally.
- One-flag CLI: Append
--hostedto anyprime eval runcommand to offload execution to Prime's platform. - Hundreds of environments available: The Environments Hub covers GPQA, SWE-bench Pro, HLE, AIME, IFEval, and custom business evals from companies like Zapier.
- Transparent leaderboards: All public runs publish full traces, sampling parameters, and tool calls alongside scores for reproducibility.
- Pricing tied to inference only: Sandbox runtime is free; you pay for model tokens via Prime Inference or your own OpenAI-compatible endpoint.
- Part of a full training loop: Hosted Evaluations integrates with Lab's hosted RL training and inference to close the evaluate-train-deploy cycle.
Running a serious LLM benchmark used to mean wrangling harnesses, spinning up sandboxes, babysitting hours of parallel compute, and debugging infrastructure failures before you even got to the results. Prime Intellect's Hosted Evaluations collapses that entire stack into a model selection and a single CLI flag.
Prime Intellect is the team behind INTELLECT-3, a 106B-parameter open MoE model trained with large-scale RL, and the Environments Hub, a community repository for RL environments and evaluations. Since launching the Environments Hub, over 1,000 unique environments have been created by 250+ contributors, with more than 100k total environment downloads. Hosted Evaluations is the natural next step: take that library and make it runnable by anyone, without any local setup.
Why evals became an infra nightmare
Evaluations are the cornerstone of LLM research and deployment. They tell you what a model can and, more importantly, cannot do. Without them, you are stuck with vibes, single-prompt tests, and cherry-picked examples from model vendors.
The problem is that the nature of evaluations has fundamentally changed. Early benchmarks like MMLU-Pro were simple: feed a multiple-choice question, score with a regex, done. But modern evals look nothing like that.
The shift from knowledge-based single-prompt tasks to multi-turn agentic coding tasks massively increased infrastructure requirements. Models inside harnesses now run for hours on a single task, editing and running code in sandboxes. To evaluate efficiently, hundreds of sandboxes need to be spun up in parallel. These infrastructure requirements are often overlooked, but become a huge bottleneck in research.

Evaluating a standard LLM is already non-trivial, but agentic systems introduce layers of complexity that traditional one-shot evaluation pipelines are not architected to handle. Because agents are stateful, active participants traversing codebases, executing terminal commands, and iterating, the challenges are less about methodology and more about infrastructure bottlenecks.
One flag to rule them all
Hosted Evaluations gives everyone access to scalable infrastructure to test models on their own evaluations across the hundreds of community-made environments on the Environments Hub. The workflow is intentionally minimal. From the dashboard, you pick an environment and a model. From the CLI, you append --hosted to your existing eval command:
prime eval run primeintellect/gpqa --model openai/gpt-oss-120b --hostedYou can also define multi-environment runs in a TOML config:
[[eval]]
env_id = "primeintellect/gpqa"
model = "openai/gpt-oss-120b"You can run evaluations locally or launch them as hosted evaluations on the platform with --hosted. The CLI supports log streaming with --follow, configurable timeouts up to 24 hours, and the ability to inject secrets for environments that need API access to external tools.

What's actually in the box
The Environments Hub already covers a wide spectrum of evaluation types. Here is a breakdown of what is available:
- Knowledge and reasoning: GPQA Diamond (graduate-level biology, physics, chemistry), MMLU-Pro, HLE, FrontierScience, AIME, APEX-shortlist
- Instruction following and tool use: IFEval, tau3-bench, GraphWalks (long context)
- Agentic coding: SWE-bench Pro, FrontierSWE, Terminalbench 2
- Custom business evals: Zapier's AutomationBench and other company-specific environments
Every run on a public environment gets published to a leaderboard, and every leaderboard entry exposes the full trace: sampling parameters, tool calls, intermediate outputs, and final scores. As evaluations become more agentic, Prime Intellect is working on Verifiers v1, their open-source library powering all the environments, to make it easier to evaluate different combinations of models and harnesses.

Pricing and the billing model
The sandbox runtime for hosted evaluations is not billed separately. Costs come entirely from model inference. Two modes are supported:
- Prime Inference (default): Uses Prime's own inference pricing for the selected model. No extra sandbox charge.
- Custom OpenAI-compatible endpoint: Pass
--api-base-urlto route model calls to your own provider. Prime bills only the sandbox compute; your external provider bills the tokens.
Total cost scales with the model you pick, prompt and completion token volume, and the product of num_examples × rollouts_per_example. The UI shows a cost estimate before you launch, so there are no surprises.
The bigger picture: closing the training loop
Hosted Evaluations does not exist in isolation. Prime Intellect's Lab platform connects environments, evaluations, hosted RL training, adapter deployments, and inference into one loop. The intended workflow is: evaluate a baseline, train a LoRA adapter with RL on the same environment, deploy it, evaluate again, repeat.
The Prime Intellect stack is uniquely suited to fill this gap: due to their experience with their hosted training offering Lab, they found solutions to all these infrastructure challenges and can leverage them for evaluations. That matters because the same sandboxing and orchestration machinery that makes RL training work at scale is exactly what agentic evals need.
For teams that want to publish their own evaluations, the path is open too. While the Environments Hub is already one of the biggest platforms for evaluations, Prime Intellect is always interested in new and exciting evaluations, whether from academic researchers who want broader reach for a benchmark, or businesses like Ramp and Zapier that want to run evaluations against their own use cases. The official documentation covers the full CLI reference and dashboard walkthrough. The Verifiers library powering the environments is open source.
The real shift here is positioning evaluation as a service rather than a project. If the friction of running a serious benchmark drops to near zero, the expectation that every model release comes with reproducible, third-party-verifiable numbers becomes a lot more reasonable.