On October 19, 2026, OpenAI published a report on the performance of language models in rare programming languages and released the MultiLangBench benchmark on GitHub under the Apache-2.0 license. In this test, a model without internet access must write a database engine compatible with SQLite 3.40.1 from scratch, and the correctness of the implementation is verified by actual SQLite using 24,000 tests adapted from its standard TCL suite. At the maximum reasoning level, GPT-6 Astra passed an average of 87.4% of checks across 20 languages, including COBOL and Fortran, compared to 26.8% for the previous generation GPT-5.6 Sol. It also emerged that the closed code migration test from COBOL is nearly exhausted: GPT-6.1 Sol scored 97.9% on it at a high reasoning level, meaning that stricter evaluations will be needed to distinguish strong models.

image

What happened

On October 19, 2026, OpenAI released the report “Advancing coding in low-resource programming languages” and simultaneously published the MultiLangBench benchmark in the openai/MultiLangBench repository under the Apache-2.0 license. The task is deterministic: a model without internet access writes a database engine compatible with SQLite 3.40.1 from scratch, and the implementation is verified by actual SQLite using 24,000 tests adapted from its own TCL suite; a sample of 272 tests is used for the final evaluation. At the maximum reasoning level, GPT-6 Astra passed an average of 87.4% of tests across 20 languages—including COBOL, Fortran, OCaml, and Visual Basic—while GPT-5.6 Sol showed only 26.8%. The gap is of a similar order for individual languages: 72.1% versus 10.9% on COBOL, and 83.5% versus 21.4% on Fortran. A separate closed migration test—porting code from COBOL to C++, Java, Kotlin, Python, or Rust with byte-for-byte output matching—yielded 86.6% for GPT-6.1 Sol at a low reasoning level and 97.9% at a high level, while GPT-5.6 Sol scored 5.4%; OpenAI acknowledged that this test is close to the ceiling and ceases to distinguish strong models further.

Context

Rare, “low-resource” languages are not museum exoticisms but working infrastructure. According to IBM data cited by OpenAI, COBOL accounts for 95% of ATM transactions and 80% of card payments, while Fortran is used in weather forecasting and nuclear reactor safety calculations. There are fewer engineers capable of reading and modifying such code, migration costs are rising, and a single error can paralyze a payment system. Prior to this release, public benchmarks testing model performance in such languages practically did not exist. The methodological peculiarity of MultiLangBench lies precisely in this: code correctness is confirmed not by a judge language model or expert evaluation, but by actual SQLite using tens of thousands of functional tests, which significantly narrows the room for interpreting results in favor of the vendor.

Why this matters for the industry

For the industry, the significance of the release lies not in the GPT-6 record but in a shift in the legacy market structure. The ability of models to write and port code in COBOL and Fortran, verified by SQLite tests, undermines the foundation of the familiar “rare specialists plus high margin” model and reduces the cost of migrating critical systems. Simultaneously, the model layer itself becomes cheaper: GPT-6 Luna inference is approximately 78% cheaper—$0.77 versus $3.48—making the migration agent pilot scheme economically sensible, where a cheap low-reasoning pass is combined with an expensive high-reasoning pass for complex cases. The open-source MultiLangBench under Apache-2.0 is also a ready-made pattern for evals teams: the agent generates code, and a real test verifies it, not a subjective judge, so the repository can be integrated into one’s own evaluation suite to compare models in rare languages independently of the vendor. For integrators with legacy practices, this is a signal to reconsider their positioning before competitors do.

Why this matters for users

For readers and developers, the event provides three practical guidelines. First: the benchmark is in an open repository and can be cloned and run on models from one’s own stack, providing an independent baseline for rare languages instead of relying on a press release. Second: a strong model is not yet a ready-made migrator; results on individual languages are noticeably lower than the average, and the benchmark measures one specific task, so trusting the automatic migration of industrial systems with data today is not possible—at most, pilots on auxiliary tasks like rewriting individual modules or generating tests. Third: the closed migration test is nearly exhausted, so third-party runs of MultiLangBench and the emergence of stricter evaluations should be expected—it is meaningful to compare models for real projects based on independent reproductions rather than vendor figures.

What is still unknown / limitations

The final evaluation of the benchmark is based on a sample of 272 tests out of 24,000 available, and the methodology for forming this sample is not disclosed in the sources—without it, it is impossible to distinguish a representative slice from a selected one; this is a caveat, not an accusation. The key generational comparison was made at the maximum reasoning level, so from open materials, it is not possible to separate the contribution of training itself from the contribution of inference-stage computations. The closed migration test has not been published, and its methodology cannot be verified externally. The benchmark itself measures one deterministic task—writing an engine compatible with SQLite 3.40.1 from scratch—and not the industrial migration of real systems with data and live integrations. Sources do not report on commercial migration deployments, and on COBOL, a significant portion of checks still fail.

Sources

Author

Look at AI, editorial team