Startup Xplore published RWE-bench (Real World Engineering Bench) — a closed benchmark consisting of 20 engineering tasks: 9 in mechanics, 5 in electronics, and 6 in aviation. The model must produce a working design, not just text about it, in a sandbox equipped with CadQuery, Gmsh, CalculiX, and Python. Verification is fully automatic; a deterministic verifier recalculates every requirement using physical solvers, without human intervention or an LLM judge. The best average result was 37% by GPT-6 Astra at a cost of $58.83 per run, while four out of twenty tasks remained unsolved by any of the eleven models across 1,100 runs.
What happened
In the test, models act as engineers: they must design a solid crankshaft for a three-cylinder diesel engine at 2,400 rpm, route a 32-bit DDR3L bus from a Zynq-7000 on an eight-layer board, or perform a CFD calculation of turbine blade flow. The verifier checks the design itself, not the text: stresses, stiffness, timing tolerances, and CFD convergence are recalculated, and an error in a single parameter results in a failed task. Across 1,100 runs, the 11 models showed the following results: GPT-6 Astra achieved 37% at $58.83 per run, Claude Opus 5.5 achieved 29% at $57.52, and GPT-6.1 Sol achieved 25% at $10.28. The bottom of the list includes DeepSeek V4.1 Flash, Qwen3.8 Max, GLM-5.3, Muse Wonder, and Grok 4.7, which had no tasks counted as solved.
Context
The key methodological idea is to remove the LLM judge from evaluation. A physical solver provides a deterministic binary verdict for each requirement, eliminating the subjectivity and hinting characteristic of language model judging and making comparisons reproducible. RWE-bench itself is closed: tasks are not provided for training, and testing third-party models is possible only by agreement with Xplore. A parallel open branch has already emerged: a team led by Tsinghua University released the EngiWorld benchmark with 1,301 open tasks, accompanied by an arXiv paper (arXiv:2609.37686), code, and datasets on Hugging Face, where a reproducible baseline for Claude Opus 5 was recorded at 44.7 out of 100.
Why this matters for the industry
For the AI industry, the logic of evaluation and procurement is changing: one must calculate not just model rankings, but the cost of a successful run, because the cost of error in engineering tasks is measurable. GPT-6.1 Sol delivers 25% at a price 5.7 times lower than GPT-6 Astra, while a failed run of the top two models costs $57–59. Products based on engineering agents are currently viable not around a "smart model," but around deterministic verification: a physical solver acting as a verifier turns unreliable LLM inference into a "design — recalculate — correct" cycle, and such a result can already be sold as "accepted by the verifier." Within a six-month horizon, this pattern will likely be replicated: follow-up benchmarks in other engineering domains, physical solver plugins for agent frameworks, and vendor optimization of agents for simulation checks are expected. Autonomous automation has not yet arrived, so pilots of "fully automatic engineering" should reasonably be repackaged into human-in-the-loop processes, where the agent generates and iterates options, while the decision and responsibility for approval remain with the engineer. In two years, based on this data, it is more realistic to expect the automation of routine tasks between engineers—such as generating initial design variants, routing standard nodes, and running parametric sweeps—rather than the displacement of engineers.
Why this matters for users
The practical takeaway for readers choosing a model for engineering tasks is that the current optimum for price and quality is GPT-6.1 Sol — 25% of tasks solved for $10.28 per run, rather than flagships where an attempt costs over $57. Cheap general-purpose models are currently useless in this narrow domain: DeepSeek V4.1 Flash at $1.03 per run yields no counted solutions, meaning savings on the model directly translate to zero results. The formula "the model designs like an engineer" is currently best considered marketing: even the best agent passes less than half of the tasks. Users can extract value today without access to the closed benchmark: the open stack (CadQuery, Gmsh, CalculiX, Python) and the open EngiWorld with code and datasets on Hugging Face allow building automatic design verification against tolerances before handing work over to a human.
What is still unknown / limitations
The main limitations are related to the benchmark design. The sample size is only 20 tasks with manual selection: the gap between 37% for GPT-6 Astra and 29% for Claude Opus 5.5 is a difference of a few tasks, and without a per-task breakdown with confidence intervals across 1,100 runs, the ranking of the top 3 is statistically unstable. The task authors are Xplore's own engineers, who sell training environments based on these same tasks, and the set is closed to independent audit, so the risk of biased selection cannot be ruled out. Finally, conclusions about the replication of the methodology and the shift to hybrid processes within a six-month and two-year horizon are forecasts that need to be verified by new measurements, not established facts.
Sources
- RWE-bench: Real World Engineering Benchmark — Xplore
- EngiWorld — Professional Engineering Agent Benchmark (Tsinghua-led team)
- EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments? (arXiv:2609.37686)
Author
Look at AI, editorial team