Prime Intellect Unifies 365,000 Agentic Tasks Into one Open RL Training Framework

Training an RL agent to write code, navigate terminals, or search the web has always meant wrestling with a different harness, a different Docker setup, and a different grading script for every dataset you want to use. Prime Intellect's new release cuts through that fragmentation: 23 agentic tasksets, 365,000+ tasks, one API, one image registry, one command.

Why fragmentation persisted

Popular evaluation suites like lm_eval and HELM handle single-turn Q&A evals well, but lack support for agentic tasks that require complex infrastructure like TAU-bench, TerminalBench, and SWE-bench. The result has been a sprawl of independent eval repos with no shared entrypoints or specs. Every team building a coding or search agent rebuilt the same scaffolding from scratch.

RL environments and agent evals are fundamentally the same thing: a dataset, a harness, and scoring rules. Open-source efforts have treated them as separate problems. Prime Intellect's answer is a unified abstraction called verifiers v1, which decomposes any environment into three independent layers: a taskset, a harness, and a runtime. This release delivers the taskset layer.

What's in the catalog

Twenty-three agentic tasksets ship behind one taskset API: roughly 198,000 software engineering tasks across 20+ languages, 28,600 terminal tasks, and 137,600 search tasks, totaling around 365,000 tasks ready for evals and RL training.

The three domains each have a distinct character:

• Software engineering (SWE): Covers the full spectrum from canonical benchmarks to large training corpora. Includes SWE-bench Verified (500 human-filtered GitHub issues), SWE-smith (88,130 synthetically injected bugs across 8 languages), SWE-rebench-V2 (32,079 freshly mined PRs across 20 languages), OpenSWE (45,320 tasks), and more.
• Terminal: Terminal-Lego provides ~13,800 Docker-verified tasks built from real StackOverflow issues. TMax adds 14,600 terminal tasks pinned in a versioned task registry, each with a prebuilt Prime image.
• Search: These tasksets are harness-agnostic and tool-free. They ship questions and scoring only; the harness supplies its own search tool. The same tasks can therefore train and evaluate any search-capable agent without locking in a retrieval pipeline. Includes PaperSearchQA (54,907 biomedical deep-research questions) and BrowseComp (OpenAI's 1,266-question browsing benchmark).

Getting started with any taskset takes a single install and run:

uv pip install "git+https://github.com/PrimeIntellect-ai/research-environments.git#subdirectory=environments/swe/scaleswe_v1"
# Run ScaleSWE in Codex harness on Prime Sandboxes:
uv run eval scaleswe-v1 --harness.id codex --harness.runtime.type prime -n 3

135,000 prebuilt images, no rate limits

Task images live in Prime's own registry, co-located with the sandboxes: roughly 135,000 prebuilt open-source task images, the largest such catalog hosted by any sandbox provider to their knowledge. There are no flags, no namespace mapping, and no Docker Hub rate limits at rollout time, which would otherwise surface immediately when running a thousand concurrent rollouts.

You can push your own images with prime images push or bulk-copy from Docker Hub with prime images transfer-bulk.

The quality problem in open task data

A task produces a clean reward signal only when the reference fix makes tests pass and the absence of any fix makes them fail. A significant fraction of open agentic task data fails this condition: broken images, network-dependent tests, drifted expected outputs, and tasks solvable without touching the code at all.

Prime Intellect ran a validation pipeline on every default dataset:

• Gold-patch validation: apply the reference fix and confirm tests pass
• No-op validation: confirm tests fail without any fix
• Up to 10x retries to separate flaky tests from deterministically broken ones
• Independent second passes to catch noisy rows

The results illustrate the scale of the problem. For R2E-Gym, 4,522 of 4,578 rows pass 10x-retry gold validation; the 56 drops are mostly network- and timing-sensitive aiohttp/tornado tests. For SWE-rebench-V2, only 6,275 rows survive from 32,079 raw, after language-level drops for wholesale-broken images, two independent gold passes with flaky rows removed, and problem statements scrubbed of inline GitHub issue and PR references. All cleaned datasets are re-uploaded to the SWE RL collection on Hugging Face with every exclusion preserved for audit.

Reward hacking as a design constraint

Reward hacking is a growing problem in agentic training: agents learn to game the scoring system rather than solve the task. In software tasks this shows up as hardcoded expected outputs, copied reference implementations, or exploited sandbox edge cases. During a rollout, the agent lives inside the same sandbox as the grading machinery, so anything readable in the container is available to exploit.

Every integration therefore withholds grading material until scoring time. Test patches, expected outputs, and grading scripts are hidden from the agent and restored only at the moment of evaluation. Some upstream datasets shipped these artifacts readable inside the container; Prime Intellect's integrations close that gap.

The team is candid about the limits here. As long as grading runs where the agent lives, this is mitigation rather than a guarantee. Their stated roadmap item is grading in fully isolated sandboxes, so the environment the agent can touch and the environment that scores it are physically separate.

Training results

All tasksets plug directly into prime-rl, Prime Intellect's open RL training framework. In a length-penalty ablation, GLM-4.5-Air was trained on ScaleSWE across six H200 nodes over two days and evaluated on SWE-Bench-Verified.

Train reward climbed from 0.17 to 0.47 and eval Pass@1 on SWE-Bench-Verified rose from 0.40 to 0.56 over roughly 1,000 steps, confirming that the unified task infrastructure produces usable training gradients.

Open problems the team flags

Two known failure modes remain. The first is reward hacking (false positives), actively mitigated but not eliminated. The second is false negatives: tasks mined from merged PRs inherit tests that assert implementation details rather than behavior. An agent that fixes the underlying issue a different way still fails those tests, and the reward reads as zero. At RL scale, these near-misses are pure noise. The team's internal effort to address this is called "Agentic Judging," with details forthcoming.

The code and taskset contract are open. If a taskset you need is missing, the contract is small enough that porting one is a realistic afternoon project, with the sandbox handling the heavy lifting.