Kaggle's AgenticRL-Bench Reveals Top AI Agents Solve Only 22.8% of Broken RL Systems
Kaggle and Eigent AI launch a benchmark where coding agents debug real RL training stacks, and the top model clears just 22.8 percent of tasks.
Takeaways
- Kaggle and Eigent AI launched AgenticRL-Bench, testing coding agents on real RL stack incidents.
- Built from 22 post-training RL tasks; 19 verifiable tasks score the public leaderboard.
- GPT-6.1 Sol leads at 22.8% pass@1, followed by Gemini 4 Carbon at 19.3%.
- 12 of the 19 verifiable tasks are solved by no agent across all attempts.
- Each agent gets 3 attempts, 8-hour budget, run on Modal with hidden checks.
- Open source with full logs: github.com/eigent-ai/agenticrl-bench.
Kaggle and Eigent AI have released AgenticRL-Bench, an evaluation suite built around infrastructure-heavy incidents from reinforcement learning systems. Coding agents must diagnose and patch failures across frameworks including verl, SGLang, Megatron, slime, Miles, and Harbor. The leading agent, GPT-6.1 Sol, records a 22.8% pass@1 score.
Post-training RL refines pretrained models with reward signals, leaving its tooling to coordinate generation, scoring, optimization, distributed compute, and checkpoints. Failures can waste long GPU runs, corrupt rewards, or produce results that cannot be reproduced. AgenticRL-Bench measures whether coding agents can repair that operational layer.
Inside the 22-task suite
The suite contains 22 post-training RL tasks, including 19 verifiable tasks scored on the Kaggle leaderboard. Titles such as Verl Upgrade Gate, Reward Determinism under Load, Trial Fault Attribution, and SQL Partial Credit Exploits reflect failures involving upgrades, concurrency, experiment tracking, and evaluation integrity.
Each agent receives three independent attempts per verifiable task. Every attempt runs headlessly as a one-shot job with an eight-hour wall-clock limit, and a task attempt passes only when every hidden check succeeds. Kaggle executes the runs on Modal to keep the environment consistent.
- Scored tasks: 19
- Attempts per task: 3
- Total scored attempts: 57
- Primary metric: Successful attempts divided by 57
- Tiebreaker: Head-to-head Elo
Three open-ended tasks extend the evaluation beyond deterministic checks. Each provides four NVIDIA H200 GPUs for 10 to 16 hours, starts from a working Group Relative Policy Optimization setup on Miles, and allows one designated intervention. A separate environment grades the resulting submission. These tasks remain outside the public leaderboard score, although their complete run logs are available.
A leaderboard with large unsolved regions
| Rank | Agent | Pass@1 |
|---|---|---|
| 1 | GPT-6.1 Sol | 22.8% |
| 2 | Gemini 4 Carbon | 19.3% |
| 3 | GPT-6 Astra | 14.0% |
| 3 | Claude Opus 5.5 | 14.0% |
Under the benchmark’s all-checks-must-pass rule, 12 of the 19 scored tasks remain unsolved by every evaluated agent. The result shows that current coding agents struggle to repair RL infrastructure reliably, even when they receive multiple attempts and substantial runtime.
Scores combine code, policy, and speed
Kaggle reports that seven of the 22 tasks trigger safety classifiers used by frontier model providers. Refusal rates vary across providers, and a blocked attempt counts as a failure. Leaderboard scores therefore reflect provider policy and deployment configuration alongside debugging ability.
An attempt can also fail by exhausting its eight-hour limit before producing a patch. Agents that spend heavily on tool calls, repeated inspection, or unproductive debugging loops may time out even when their underlying model can identify the defect.
Deeper than issue resolution
SWE-bench evaluates an agent’s ability to resolve GitHub issues in software repositories. MLE-bench evaluates agents through Kaggle machine-learning competitions; its best reported setup, o1-preview paired with AIDE, earns a medal in 16.9% of competitions on average. AgenticRL-Bench targets the lower-level systems work required to keep post-training pipelines operational.
Current RL stacks combine training, inference, reward computation, scheduling, storage, and experiment tracking across several projects. A defect may emerge only under distributed load or through an interaction between components, making diagnosis dependent on logs, runtime behavior, configuration, and source code.
Public artifacts support analysis beyond the aggregate scores. The repository includes raw logs and attempt-level results for Claude Opus 5.5, GPT-6 Astra, Gemini 4 Argon, Gemini 4 Carbon, Grok 4.7, and Muse Wonder Full 1.0, plus all three attempts for each open-ended task.
How teams can use the benchmark
Teams building coding agents can use AgenticRL-Bench as a controlled test of several production-relevant behaviors:
- Regression testing: Measure whether changes to prompts, tools, memory, or orchestration improve per-attempt success.
- Failure analysis: Use raw logs to separate incorrect patches, safety refusals, timeouts, and stalled investigations.
- Retry evaluation: Compare outcomes across three independent attempts instead of relying on a single run.
- Tool design: Identify where agents need better access to logs, tests, dependency graphs, or distributed-system state.
- Deployment testing: Quantify how provider policies and runtime limits affect usable capability.
A best pass@1 score of 22.8% leaves current agents far from unattended RL infrastructure maintenance on this suite. The logs and hidden-check outcomes provide concrete data for improving agent scaffolding, debugging tools, and RL-specific training.