The open benchmarking initiative aistack, created by the imec research center, published a comparison of three coding-agent harnesses — Claude Code, Codex, and Pi — on two open models: Qwen3.8-27B (FP8, one NVIDIA H200) and GLM-5.3-Flash (FP8, four H200s). On 64 SWE-Bench Pro tasks, all six harness-and-model combinations showed a resolve rate of 44% to 53%, meaning accuracy was practically independent of the harness choice. But the cost differed by almost twofold: the GPU cost of a full run was $22.8 for the Codex + GLM-5.3-Flash combination versus $45 for Claude Code and Pi.

image

What happened

aistack ran three coding-agent harnesses — Claude Code, Codex, and Pi — on two models: Qwen3.8-27B in FP8 format on one NVIDIA H200 and GLM-5.3-Flash in FP8 on four H200s. The set of 64 tasks was selected from SWE-Bench Pro, and each harness worked with each model, yielding six combinations. The resolve rate in all six cases fell within the 44–53% range — the spread was only 2–3 tasks, meaning the harness choice had almost no effect on accuracy. The difference showed up in resources: Codex with GLM-5.3-Flash generated about 2.1 million output tokens versus 3–4 million for the other combinations, and it required 192 million input tokens on GLM versus 480 million for Claude Code. The final GPU cost of the 64-task run was $22.8 for Codex with GLM-5.3-Flash versus $45 for Claude Code and Pi. The measurement recorded two anomalies: in Pi, workers got stuck in a repeating loop (a “doom loop”), the model degraded into nonsense, and the run was killed by the orchestrator at the 4-hour limit; in Claude Code, version v2.1.233 broke prefix caching — the average hit rate dropped from 97.36% to 29.84%, and per-request latency increased roughly tenfold.

Context

A harness is the agent wrapper that manages context, tools, and agent stopping on top of an LLM; until recently, its impact on results was rarely measured, and the choice mostly came down to model selection. The novelty of the aistack measurement is methodological: the harness variable is isolated for the first time from the model variable, and economic metrics — token efficiency, wall time, and run cost — are added to accuracy. Both participating models run in standalone self-hosted inference on NVIDIA H200 GPUs in FP8 quantization, so the numbers reflect a real operational picture rather than cloud API prices. The separate case with Claude Code v2.1.233 is known from a GitHub issue about latency: a point release of the harness broke prefix caching — the caching of KV suffixes for the static part of the context, which affects both latency and token spend.

Why this matters for the industry

For builders and companies operating agents, the imec measurement provides a measurable argument: the choice of harness is comparable in impact to the choice of model, because at an equal accuracy of 44–53% on SWE-Bench Pro, the cost of a 64-task run varies from $22.8 to $45 due to token efficiency and wall time. This directly determines the unit economics of agent products and self-hosted inference: what needs to be optimized is context engineering and prefix caching, not just LLM selection. The competitive advantage of agent products shifts from access to a model toward the quality of the harness — token budgets, observability, and release stability. The Claude Code v2.1.233 regression additionally shows that a single point version of a harness can collapse the prefix-cache hit rate and multiply latency, so it makes sense for teams to pin versions, run canary runs before updating, and monitor cost per task.

Why this matters for users

For those who run inference themselves and operate coding agents, the measurement provides a ready-made checklist for auditing their own stack: token efficiency (the ratio of output to input tokens), prefix-cache hit rate, wall time, and KV-cache load. The real case with the Claude Code v2.1.233 release shows why you should not update harness versions in production blindly: the drop in cache hit rate from 97.36% to 29.84% multiplied the latency of every request. The practical step is to add monitoring of prefix-cache hit rate and cost per resolved task, and to test any new harness version on a small canary run before rolling it out to the production environment.

What is still unknown / limitations

Accuracy across all combinations remains at a low absolute level: both models solve fewer than half of the SWE-Bench Pro tasks, so the equality of results does not mean that accuracy has become a threshold “that no one discusses anymore.” With 64 tasks, one task is about 1.6 percentage points, and a spread of 2–3 tasks does not allow statistically confirming either the presence or the absence of a systematic difference between harnesses. The two anomalies — the “doom loop” in Pi and the broken prefix caching in Claude Code v2.1.233 — indicate the fragility of the measurement itself: the result is sensitive to harness versions and orchestrator behavior. Finally, the measurement was done on a single hardware configuration and two specific models, and the market response so far is weak, so the conclusions should be transferred to other models, tasks, and pricing with caution.

Sources

Author

Look at AI, editorial team