OpenAI, in collaboration with biotech company Ligo, introduced BioBasicsBench—a public benchmark of 100 tasks on routine 'wet' lab work where an AI agent must interpret raw data from real experiments: gels and blots, protocols, instrument tables, microscopy, videos, and photographs. At the maximum reasoning level xhigh, GPT-6 Astra showed the best result—66.7% average rubric score proportion, while other tested models scored between 31.2% and 62.0%. Materials necessary for independent verification have not yet been disclosed.

image
image

What Happened

The benchmark covers five areas of routine 'wet' lab work: protein methods, microscopy and cytometry, quantitative analysis, nucleic acids, and molecular cloning. The agent is given not quiz questions, but raw experimental artifacts that need to be read and interpreted: gels and blots (31 tasks), protocols and lab notes (27), instrument tables and exports (20), desk photos (18), microscopy (14), instrument screens (13), sequencing files (12), protocol videos (7), and Petri dishes (7). At the maximum reasoning level xhigh, the results were distributed as follows: GPT-6 Astra—66.7% with a 95% confidence interval of 60.3–73.1%, GPT-6.1 Sol—62.0%, GPT-5.6 Sol—43.8%, GPT-6 Luna—38.3%, GPT-5.5—35.4%, GPT-5.4—33.2%, GPT-5.2—31.2%. A total of 18,500 results were collected: 37 configurations on 100 tasks with 5 attempts each. The score is calculated as the average proportion of rubric points with partial credit, not as the proportion of fully solved tasks.

Context

Until now, agent work with raw lab data remained an internal metric for companies: public tests checked biological knowledge, not the ability to read materials from real experiments. BioBasicsBench is the first to do this openly: tasks include experiment failures and situations with insufficient evidence, and strong configurations demonstrate a pattern of checking data sufficiency—for example, refusing to draw a conclusion when fluorescent channels overlap and an unambiguous answer is impossible. The gap between model generations is sharp: the two strongest configurations approach the two-thirds threshold of possible points, while the others stay closer to one-third. Native multimodality—the ability to directly read photos, videos, and instrument screens—is discussed as an explanation; this is an interpretation of the observed gap, not a factor isolated within the benchmark.

Why This Matters for the Industry

For agent developers, the evaluation methodology here is almost more valuable than the final numbers: five attempts per task to measure variability, rubrics with partial credit, and cases with corrupted data such as incorrect control labels and tasks where the correct behavior is not to draw a conclusion without sufficient evidence. This template can be transferred to proprietary test sets: including 'raw' formats like photos, instrument screenshots, and protocol videos, and measuring cost of inference in output tokens per task alongside accuracy. The cost signal is notable: GPT-6 Astra at xhigh with 66.7% spends 4,704 output tokens per task compared to 9,099 for GPT-5.6 Sol, meaning approximately 48% less consumption with a higher score. The benchmark itself is not a product and there is nothing to deploy based on it, but the 100 tasks provide a free map of which lab operations—from reading gels to parsing instrument exports—agents are already capable of interpreting.

Why This Matters for Users

The BioBasicsBench leaderboard is convenient to track as an indicator of AI agent maturity for science: model updates can now be compared by their ability to work with real experimental data, not just conversational skills. The tasks themselves suggest which lab routines may eventually move to AI assistants: decoding gels and blots, automatic checking of protocols and metadata, and reading instrument exports. The realistic role of such agents today is an assistant with mandatory human verification, not an autonomous lab technician. A probable, but currently only predictive scenario: if OpenAI opens the benchmark materials, independent runs of competing models and its replicas for chemistry and materials science will appear, and the first practical products may be automatic checking of experiment protocols and metadata and assistance in interpreting gels.

What Is Still Unknown / Limitations

Full rubrics, raw data, and code have not been published, so independent replication of results is currently impossible. The leader's 95% confidence interval width of ±6.4 percentage points means that GPT-6.1 Sol's 4.7-point lag based on available data is not statistically evaluated—intervals for other models are not provided in the source. The efficiency comparison covers a single pair of configurations, GPT-6 Astra and GPT-5.6 Sol at xhigh, and only output tokens, so the conclusion that 'stronger means cheaper' cannot be extrapolated to the entire market. Finally, the headline 66.7% should not be read as 'two-thirds of the lab is automated': this is partial credit on rubrics, not the proportion of fully solved experiments, and scenarios regarding code publication and replicas for other sciences are predictions, not established facts.

Sources

Author

Look at AI, editorial team