Training an RL agent to write code, navigate terminals, or search the web has always meant wrestling with a different harness, a different Docker setup, and a different grading script for every dataset you want to use. Prime Intellect's new release cuts through that fragmentation: 23 agentic tasksets, 365,000+ tasks, one API, one image registry, one command.
The fragmentation problem nobody wanted to solve
Popular evaluation suites like lm_eval and HELM offer convenient entrypoints into single-turn Q&A evals, but generally lack support for tasks that are agentic in nature or require complex infrastructure setups like TAU-bench, TerminalBench, and SWE-bench, resulting in a proliferation of independent eval repos without shared entrypoints or specs. Every team building a coding or search agent was essentially rebuilding the same scaffolding from scratch.
RL environments and agent evals are fundamentally the same thing -- dataset plus harness plus scoring rules -- but current open-source efforts generally treat them as separate. Prime Intellect's answer is a unified abstraction called verifiers v1, which decomposes any environment into three independent layers: a taskset, a harness, and a runtime. The new release is the taskset piece of that story, and it's substantial.
What's actually in the box
The catalog ships first-party integrations for 23 agentic tasksets behind one taskset API -- roughly 198,000 software engineering tasks across 20+ languages, 28,600 terminal tasks, and 137,600 search tasks, totaling around 365,000 tasks ready for evals and RL training.
The three domains each have a distinct character:
- Software engineering (SWE): Covers the full spectrum from canonical benchmarks to massive training corpora. Includes SWE-bench Verified (500 human-filtered GitHub issues), SWE-smith (88,130 synthetically injected bugs across 8 languages), SWE-rebench-V2 (32,079 freshly mined PRs across 20 languages), OpenSWE (45,320 tasks), and more.
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves