The Arena.ai team published the study “HarnessTax: How Much Does the Harness Matter for Coding Agents?”: across 21 “model × harness” pairs, the choice of agent harness shifted the share of solved tasks by only ±2% on SWE-bench Lite and ±5% on Terminal-Bench 2.0, while the cost per task varied by up to 5x. The same model in the minimalist Pi harness solved tasks with nearly the same success as in Claude Code, but at roughly half the cost. The authors conclude that comparing coding agents without specifying the harness is incorrect, and that harness complexity is an empirical trade-off, not a default value.

image
image

What happened

On September 16, 2026, the Arena.ai team consisting of M. Z. Pan, S. Yang, N. Arabzadeh, W.-L. Chiang, I. Stoica, and M. Zaharia published the paper “HarnessTax: How Much Does the Harness Matter for Coding Agents?”. The authors ran 7 models in three agent harnesses — Claude Code, Codex CLI, and the minimalist Pi — and obtained 21 “model × harness” pairs, which they tested on SWE-bench Lite and Terminal-Bench 2.0: 30 tasks per benchmark, 3 repeats, up to 100 agent steps, with 95% confidence intervals on 10,000 bootstrap resamples. In combination with Claude Fable 5, the harness changed success from 97.8% in Claude Code to 96.7% in Pi, but the price — from $1.33 to $0.67 per task. Betting on the “native” harness also did not pay off: in 9 of 12 comparisons for Anthropic and OpenAI models, a foreign harness gave a better result — Sonnet 4.6 showed 68.9% in Codex versus 66.7% in Claude Code, and GPT-5.6 Sol on Terminal-Bench 2.0 — 83.3% in Pi versus 78.9% in Codex at a price of $0.42 versus $0.76 per task.

Context

A harness is an agent shell that manages the model's tools and context: it determines which tools are available, how the initial context is assembled, and how the agent's steps are organized. Before this work, the harness remained a hidden confounder in all coding agent comparisons: leaderboards and reviews recorded the result of a specific “model plus default harness” combination and attributed it entirely to the model. No one had previously conducted a systematic measurement of the overpayment for the harness — the “harness tax” — separately from the model. The study suggests treating harness complexity as an empirical trade-off, not a default value: the minimalist Pi costs only four tools (read/write/edit/bash), and its initial context is more than 10 times smaller than Claude Code's.

Why this matters for the industry

For the industry, the effect is primarily methodological: before citing any coding agent result, the harness must now be specified, because the default shell introduces up to 5x variance in money with almost unchanged quality. The price is not explained by the volume of the agent's work: Claude Code and Pi have almost equal numbers of steps (15.4 versus 15.3), and the geometric mean of the price coefficient is about 2.0x, meaning the overpayment lies in the overhead costs of context and tooling, not in the number of steps. For product teams, the agent harness is not a source of quality, but a cost item that can be measured and optimized; premium prices for “branded” shells lose their justification, and minimal harnesses like Pi reach the Pareto frontier. Open profiling traces and the AgentBRANE dashboard make all 21 pairs reproducible and comparable, so multi-harness evaluation and the metric “price per solved task” can become an industry standard.

Why this matters for users

The practical takeaway for a reader who pays for a coding agent: if the model remains the same and only the shell changes, success barely changes — meaning it makes sense to try an alternative CLI and save up to 5x without losing quality. Ready-made benchmarks from the study: GPT-5.6 Luna is the cheapest option on both benchmarks, Kimi K3 with open weights stays close to the optimum in terms of price-to-quality ratio, and switching from Claude Code to Pi for the same model saves about half. Before paying for a “branded” agent, the “model × harness” pair can be tested on your own tasks: traces, graphs, and the dashboard are open at harnesstax.github.io, and the results are reproducible.

What is still unknown / limitations

The conclusion about quality parity is proven only for three specific shells (Claude Code, Codex CLI, Pi) and two benchmarks with 30 tasks and 3 repeats each: it cannot be transferred to any harnesses, long horizons, and more complex tasks. The formula “quality is bought by the model, not the shell” does not follow from these results — the data only speak of the limits of ±2% on SWE-bench Lite and ±5% on Terminal-Bench 2.0, and in other classes of tasks the role of the shell may turn out to be significant. This will have to be checked by independent replications on larger task sets; in addition, the price conclusions are tied to the current tariffs of models and shells, which may change.

Sources

Author

Look at AI, editorial team