Agent harness matters more than model: same accuracy, twice the price

The open benchmarking initiative imec aistack compared Claude Code, Codex, and Pi on Qwen3.8-27B and GLM-5.3-Flash models across 64 SWE-Bench Pro tasks, achieving identical accuracy of 44–53% across all six combinations, but different run costs — $22.8 versus $45 for Claude Code and Pi.

Compiled by Sergey KostenchukPublished 2026-09-10Updated 2026-09-18

2026-09-10 Research Anthropic

🤖 Agent harness matters more than model: same accuracy, twice the price

aistack, the open benchmarking initiative by imec, compared Claude Code, Codex, and Pi on Qwen3.8-27B and GLM-5.3-Flash models across 64 SWE-Bench Pro tasks. Accuracy across all six combinations was 44–53%: harness choice barely affects resolve rate.

🌍 The difference is in price: Codex with GLM-5.3-Flash produced ~2.1M output tokens versus 3–4M for the others, and 192M input tokens versus 480M for Claude Code. The 64-task run cost $22.8 versus $45 for Claude Code and Pi — nearly twice as expensive at equal accuracy.

👤 If you run your own inference, monitor token efficiency, prefix-cache hit rate, and wall time: in Claude Code v2.1.233, cache hit rate dropped from 97.36% to 29.84%, and Pi agents got stuck in a "doom loop" until the run was killed by the 4-hour limit. Don't update harnesses blindly.

Source 1: https://aistack.imec-int.com/blog/harness-cost Source 2: https://news.ycombinator.com/item?id=49639415

Sources