OpenAI Claims Major Leap in COBOL Modernization, But Withholds Test Details
OpenAI announced dramatic improvements in its models' ability to migrate legacy COBOL code, though the company is keeping its evaluation methodology private while releasing a separate benchmark across 20 programming languages.
OpenAI this week highlighted progress in legacy software modernization, unveiling benchmark results that point to substantial gains in its models' capacity to handle COBOL migration and software development in languages where training data is scarce. The company's latest models demonstrated marked improvements, though the specifics of one key evaluation remain undisclosed.
On OpenAI's proprietary COBOL migration test, GPT-6.1 Sol achieved 86.6% accuracy at low reasoning effort—a sharp rise from GPT-5.6 Sol's 5.4%—while consuming 39% less computational resources. When operating at higher reasoning effort, the newer model reached 97.9% accuracy.
The two evaluations OpenAI presented show uneven levels of transparency. The company has not made public the specifics of its COBOL evaluation. However, it has released MultiLangBench, a test suite available on GitHub that measures whether models can construct a SQLite-compatible database engine from the ground up across 20 programming languages.
OpenAI characterized the results as "substantial improvements across both evaluations, delivering much higher accuracy at lower cost." For teams working on legacy software maintenance and modernization, such results could prove significant. Yet without access to the public COBOL test cases, the development community cannot verify the evaluation independently.
COBOL Migration: A Persistent Challenge
According to OpenAI, its COBOL migration evaluation "tests whether a model can migrate existing programs to another language while preserving their behavior." The assessment covers migrations from COBOL to C++, Java, Kotlin, Python, and Rust.
To pass, each migration must satisfy a battery of hidden test cases demanding "byte-for-byte parity on observable outputs." GPT-6.1 Sol's substantially higher accuracy marks genuine progress, though OpenAI is not alone in applying AI to COBOL modernization.
COBOL remains a formidable challenge for AI coding tools, despite its ubiquity in critical systems. IBM reports that COBOL underpins over 40% of all online banking systems, supports 80% of in-person credit card transactions, and handles 95% of all ATM transactions. Yet models encounter less COBOL code and documentation compared to languages like Python and JavaScript. The scarcity of experienced COBOL developers compounds the difficulty, and system documentation frequently lags behind the software it describes.
Broader Performance Across Multiple Languages
OpenAI's second evaluation, MultiLangBench, gauges models' performance in building software from scratch. The benchmark "tests whether a model can build a SQLite-compatible database engine across 20 different programming languages."
The test provides a database and SQL query as input, then evaluates whether the model produces a correctly modified database. The reference SQLite implementation serves as ground truth to verify that the resulting database contains the expected data.
GPT-6 Astra, OpenAI's primary model, achieved an average 87.4% pass rate across the 20 languages—a substantial jump from GPT-5.6 Sol's 26.8%. The model scored above 83% on 19 of the 20 languages and improved on all of them.
GPT-6 Luna, the lower-cost alternative, performed less impressively with an average pass rate of 40.3%. Nevertheless, it surpassed GPT-5.6 Sol across every language tested, while carrying an estimated 78% lower API cost.
Transparency Gaps and Competitive Context
OpenAI's reported results from both its COBOL migration evaluation and MultiLangBench appear promising, yet benchmarks offer only a partial picture. As of Thursday, Real-SWE 1.1, which employs private codebases, ranked Claude Opus 5.5 first at 56.8% resolution. An earlier Real-SWE assessment covered by The New Stack showed different leadership and lower overall scores.
A critical limitation persists: developers remain unable to examine OpenAI's COBOL migration benchmark. The announcement credits input from developers and enterprise customers for shaping the effort, yet without access to the underlying tests and datasets, independent verification of the results is impossible.
OpenAI acknowledges a significant constraint, stating that "applying it to production software still requires expert review and testing against the system's business rules and dependencies."
OpenAI enters a space where competitors have already established footholds. Anthropic detailed how Claude Code could support COBOL modernization on February 23. AWS, as The New Stack previously reported, rolled out AI agents for mainframe migrations in 2024.