OpenAI's GPT-6.2 Astra Hits Record 29% on AI Self-Improvement Benchmark

Vals AI's pre-release testing shows OpenAI's newest model inching toward human-level autonomous AI research, matching references on pre-training while stumbling on post-training.

·
·
·
OpenAI's GPT-6.2 Astra Hits Record 29% on AI Self-Improvement Benchmark
Read5 min
TypeNews
TopicLlms · Agents
  • GPT-6.2 Astra scored 29.01% on the Vals RSI Index, up from 27.07% for GPT-6 Astra.
  • The benchmark is calibrated so that 50% equals human-level autonomous AI R&D performance.
  • Astra came within 1% of the compression reference and 3% of the LM training reference.
  • It beat GPT-6 Astra on compression using one-fifth the compute and half the output tokens.
  • Post-training was poor: its fine-tuned checkpoint scored 33.46%, below the 35.38% parent baseline.
  • Vals concludes recursive self-improvement at OpenAI is progressing incrementally and remains human-directed.

OpenAI’s GPT-6.2 Astra scored 29.01% in Vals AI’s pre-release testing on a modified RSI Index, up from GPT-6 Astra’s 27.07%. It is the highest result Vals has reported for an OpenAI model. The benchmark measures whether an autonomous agent can complete AI research tasks under fixed time and compute limits, with 50% calibrated as human-level performance on each task.

RSI stands for recursive self-improvement, the idea that AI systems could contribute to the research and engineering required to build more capable successors. Vals tests a bounded version of that idea through tasks involving model training, compression, evaluation systems, and post-training.

Four tasks inside a sealed sandbox

Each run places an agent in a sealed environment with a fixed budget and requires one final artifact. Vals scores that artifact on a reference-anchored logarithmic scale: 0 represents the starting baseline, 0.5 represents a published human or frontier-model result, and 1 represents the estimated theoretical optimum. The overall index is the mean of four tasks:

  • Compression: Losslessly compress 100 MB of Wikipedia within 24 hours on one H100 GPU. The score uses bits per byte and compares results with the nncp neural compressor.
  • Language-model training: Modify a nanoGPT-style program to minimize validation bits per byte, with final performance measured on a hidden test split.
  • Harness engineering: Improve an agentic judge built around Gemini 3.5 Flash-Lite that grades coding submissions.
  • Post-training: Improve a 35-billion-parameter mixture-of-experts Qwen checkpoint on a held-out finance-agent benchmark, using 30 hours on sixteen RTX PRO 6000 GPUs. A mixture-of-experts model activates selected groups of parameters for each input instead of using the full network every time.

Vals used standard agent environments for each model family: Claude and GLM ran in Claude Code, GPT in Codex, Gemini in Antigravity, Kimi in Kimi Code, Qwen in Qwen Code, Muse in Muse Code, and Grok in Grok Build.

Long experiments played to Astra’s strengths

Astra’s clearest gains came from compression and language-model training, where the objective was measurable and the agent could test many variations. It achieved 1.2582 bits per byte on enwik8, within roughly 1% of the nncp reference, while using one-fifth of GPT-6 Astra’s compute and half as many output tokens. Lower bits per byte indicate better compression.

On language-model training, Astra reached 0.8744 bits per byte, about 3% above the published 0.85 reference from a 64-layer character transformer. The agent began with an older, less efficient optimizer than competing systems, then continued finding better training recipes throughout the 24-hour budget.

Vals extended that run to 48 hours in a separate test outside the official benchmark score. Astra was still improving around hour 42 and eventually matched the published reference on held-out data. In Vals’ chart, which labels GPT-6.2 Astra as “Aurora,” competing models plateaued earlier while Astra continued reducing bits per byte.

Post-training erased performance

Astra’s post-training pipeline scored 33.46% against the parent checkpoint’s 35.38%, a regression of 1.92 percentage points. The agent pursued a specialized method informed by recent mixture-of-experts research, but the resulting model performed worse than the starting checkpoint.

Vals has observed similar post-training regressions from other OpenAI models on this task. Claude Opus 5.5 leads the category at 44.53% after identifying a specific tool-use defect and generating targeted training data to correct it.

The harness-engineering run showed a related evaluation problem. Astra tested browser-based and source-code judges, then selected a source-code approach that scored well during development but transferred poorly to unseen tasks. That gap indicates overfitting to the development set.

Automation still needs a research lead

Across the four tasks, Astra performed best when it could refine a defined recipe against a reliable metric. Its weaker results came from choosing a high-leverage direction, validating whether that direction generalized, and preventing regressions in a complex downstream system.

Vals reports that the strongest submissions mainly combined and tuned established techniques. The evaluated agents produced meaningful gains through integration and empirical search without introducing a broadly applicable new algorithm. The benchmark therefore captures bounded artifact production under human-defined goals; objective setting, research prioritization, and deployment remain outside its scope.

OpenAI’s system card describes a similar capability profile. The company says it has met an internal goal of building an automated research intern that can complete well-defined, multiday tasks under human direction. Astra’s RSI results support that classification, particularly on long-running optimization work with objective scoring.

Anthropic retains the task-level lead

On the separate, broader Vals Index, GPT-6.2 Astra scores 67.66% and ranks seventh, trailing Claude Fable 5.2 by 3.65 percentage points. The broader index and RSI Index use different evaluations, so their percentages are not directly comparable.

Anthropic models also lead the RSI compression and language-model training tables. Claude Fable 5.2 and Claude Opus 5.5 hold the strongest results on both pre-training tasks, despite Astra’s improvement over its OpenAI predecessor.

Where developers can put it to work

Astra’s results support day-long autonomous experiments when teams can define the search space, metric, baseline, and compute ceiling in advance. Suitable work includes tuning training recipes, testing compression methods, and iterating on judge prompts or evaluation code.

  • Preserve a baseline: Compare every final artifact with the untouched model, judge, or training pipeline.
  • Use hidden evaluations: Hold-out tests can expose approaches that improve development scores without transferring to new inputs.
  • Gate model updates: Require human approval and reproducible gains before deploying post-trained weights.
  • Track resource use: Record compute, elapsed time, output tokens, checkpoints, and failed experiments alongside the final score.
  • Keep rollback paths: Autonomous agents can submit regressions even when their intermediate experiments appear promising.

The 29.01% result places Astra in the role of a capable research executor rather than an independent research director. It can search established technical paths for extended periods, while human researchers continue to choose the path, verify the evidence, and decide whether the resulting artifact is safe to use.

Trending
  • No trending articles

Comments

avatar

Next Reads