OpenAI's GPT-6 Astra Cracks COBOL, Scoring 72% on Banking's Ancient Code
OpenAI's newest models jump from single-digit to near-perfect accuracy on COBOL migration and show massive gains across 20 programming languages.
- OpenAI's GPT-6 Astra hits 87.4% on MultiLangBench, up from 26.8% for GPT-5.6 Sol.
- GPT-6.1 Sol reaches 97.9% on COBOL-to-modern-language migration at high reasoning effort.
- Low-effort COBOL migration jumped from 5.4% to 86.6% accuracy at 39% lower cost.
- Evaluation requires byte-for-byte output parity, catching subtle fixed-point arithmetic bugs.
- MultiLangBench and tests released on GitHub covering 20 languages including Fortran, OCaml, Zig.
- Targets legacy banking, government, and scientific codebases where original developers are long gone.
OpenAI has published benchmark results showing large gains on low-resource programming languages, including COBOL, Fortran, OCaml, and Visual Basic. The company also released MultiLangBench, an open evaluation for comparing coding models across 20 languages, alongside a separate test for migrating COBOL programs without changing their behavior. OpenAI detailed the work in its published results.
Low-resource languages have far less public training data than Python or JavaScript, yet they remain embedded in critical systems. OpenAI cites estimates that COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions. Fortran is used in weather forecasting and nuclear record safety analysis.
Benchmark gains at a glance
MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in 20 programming languages. The migration evaluation measures whether a model can translate a COBOL program into a modern language while preserving every tested behavior.
| Evaluation | Earlier result | New result | Change |
|---|---|---|---|
| MultiLangBench average | GPT-5.6 Sol: 26.8% | GPT-6 Astra: 87.4% | +60.6 percentage points |
| COBOL | GPT-5.6 Sol: 10.9% | GPT-6 Astra: 72.1% | +61.2 points |
| OCaml | GPT-5.6 Sol: 25.6% | GPT-6 Astra: 85.8% | +60.2 points |
| Fortran | GPT-5.6 Sol: 21.4% | GPT-6 Astra: 83.5% | +62.1 points |
| Visual Basic | GPT-5.6 Sol: 27.5% | GPT-6 Astra: 89.4% | +61.9 points |
| MultiLangBench, lower-cost model | GPT-5.6 Sol: 26.8% | GPT-6 Luna: 40.3% | About 78% lower estimated API cost |
| COBOL migration, low reasoning effort | GPT-5.6 Sol: 5.4% | GPT-6.1 Sol: 86.6% | 39% lower estimated API cost |
| COBOL migration, high reasoning effort | GPT-5.6 Sol: 5.4% | GPT-6.1 Sol: 97.9% | +92.5 points |
GPT-6 Astra exceeds an 83% pass rate in 19 of the 20 languages. COBOL remains the hardest at 72.1%, still more than six times the previous model’s score. Reasoning effort is an API setting that controls how much internal computation a model uses before responding; higher settings can increase latency and token consumption.
The stated cost reductions cover estimated API use. They exclude the engineering work required for test development, dependency analysis, code review, integration, and deployment.
COBOL migrations fail on semantics
COBOL programs often encode financial and operational rules through language-specific behavior. Fixed-point arithmetic, for example, discards excess decimal digits unless the program explicitly requests rounding. A port that rounds those values can compile, pass incomplete tests, and still produce incorrect interest or settlement calculations at scale.
OpenAI counts a migration as successful only when every test passes. Hidden tests require byte-for-byte parity in observable output, creating a stricter target than benchmarks that accept code after compilation and a small set of visible assertions.
One task, 20 languages
Each MultiLangBench implementation receives an initial database and a SQL query, then produces a modified database. A reference SQLite implementation checks whether the resulting file contains the expected data.
Using database files as the interface allows one test suite to evaluate implementations written in languages as different as Zig, Common Lisp, PowerShell, and COBOL. The design avoids language-specific output formats and gives each candidate the same observable target.
MultiLangBench measures performance on a constrained database-engine task. Repository-level maintenance, framework use, dependency management, and production debugging require separate evaluations. OpenAI has released the evaluation setup, test suite, and per-language results in the benchmark repository, allowing teams to inspect the methodology and run their own comparisons.
Practical uses for legacy systems
- Migration pilots: Generate candidate COBOL-to-Java, Python, or Rust translations, then compare the original and migrated programs against identical inputs.
- Maintenance work: Explain unfamiliar routines, trace data flow, and propose patches in codebases with sparse documentation.
- Business-rule recovery: Extract behavior from existing code and encode it in tests before changing the implementation.
- Scientific computing: Assist teams maintaining Fortran code for weather, climate, physics, and engineering workloads.
Production trials need strict controls
The reported accuracy still leaves failure cases, including errors that may surface only during reconciliation, reporting, or financial close processes. Production evaluation should cover the surrounding system as well as the translated source code.
- Inventory copybooks, file encodings, job-control scripts, database calls, and external dependencies.
- Build a representative corpus of inputs and known outputs, including malformed records and historical edge cases.
- Run differential tests that execute the original and migrated programs against the same data.
- Have engineers with domain and language expertise review arithmetic, state changes, error handling, and record layouts.
- Deploy through shadow runs or staged rollouts with reconciliation and rollback procedures.
The long tail comes into range
Broader language coverage could reduce the time required to understand and modify systems that receive little benefit from models trained primarily on popular GitHub languages. The strongest near-term use is likely to be assisted analysis, test generation, and migration drafting, with engineers retaining responsibility for validation and release decisions.
OpenAI says the models are available through its API and is requesting feedback on additional languages and workflows. Teams maintaining Ada, PL/I, RPG, or similarly underrepresented codebases can use the public benchmark as a starting point for their own task-specific evaluations.
Comments
