Ben Swerdlow, co-founder of startup Freestyle, built the open Brood War Bench benchmark: 19 language model configurations in Codex, Claude Code, and Grok agent harnesses each played 171 matches against each other in StarCraft: Brood War. The leader was Codex on the Astra model with the maximum reasoning level — 18 wins with zero losses at an average cost of about $10.54 per game. However, no one rose above the novice level: the models played the real-time strategy game like a turn-based one and died while thinking.

image
image
image

What happened

The benchmark is structured as a round-robin tournament where each configuration played each other once. First place was taken by Codex on the Astra model with xhigh reasoning level, second — the same model at medium level with 16 wins, third — Claude Fable with 15, fourth — Astra at low level with 14. At the bottom of the table, results are noticeably weaker: three Grok 4.6 configurations took only three matches in total, and Claude Haiku did not win a single one. The most illustrative failure was recorded in match G043: Grok 4.6 at xhigh produced 11,138 reasoning tokens in 43 minutes, issued only six command packets, and did not build a single combat unit. All match recordings are published, so each such failure can be broken down step by step.

Context

Brood War Bench is one of the few open real-time agent benchmarks, and this is its main value for evaluating long-horizon agents. The environment lives while the model reasons: the price of an error here is measured in seconds, not tokens, as in familiar turn-based tasks. The agent acts through a harness — Codex, Claude Code, or Grok — and the model's reasoning level (xhigh, medium, or low) determines how much it allows itself to reflect before each step. The project's author, Ben Swerdlow, is the co-founder of startup Freestyle, which releases full Linux VMs for AI agents, so agent harnesses are his everyday work. An important frame for reading the table: the opponents of the configurations were only other agents, there was no human among the participants.

Why this matters for the industry

For the industry, this is a rare case where an agent benchmark reveals specific weaknesses of harnesses, not "AI weakness in general." Codex on Astra won not with macro play, but with disruption: early sending of workers to hit enemy buildings while the opponent "thinks" for tens of seconds what to do. At the same time, its subagents for economy, production, and army were almost not coordinated and sent units into battle one by one. Claude Fable, the only one that systematically built economy and technology tree — with an exit to Lair, Spire, Mutalisks, and Robotics Facility — took only third place: structurally correct play lost to timing. Grok 4.6 failed not in strategy, but in the very "observe-act" cycle. The influence of the reasoning level turned out to be non-monotonic: Astra at xhigh (100%) is stronger than Astra at medium (88.9%), but Sol at xhigh (61.1%) is worse than Sol at medium (72.2%) and Sol at low (66.7%). The outcome is determined by the harness and call economics: the cheapest strong option is Codex 5.6 Luna/low with 50% wins for just $0.42 per game, while the leading Astra/xhigh costs about $10.54. The open recordings of 171 matches are a ready-made corpus of long-horizon agent failures for those who build subagent orchestration and real-time cycles; deploying such cycles on these configurations in production is not yet possible.

Why this matters for users

Breakdowns and recordings of all matches are available at bw.swerdlow.dev/report, and you can run your own agent against the listed configurations at bw.swerdlow.dev — the project accepts third-party participants. For the reader, this is a clear calibration of expectations: agents on top models — Codex on Astra, Claude Opus 5 with Fable, Grok 4.6 — really break down in real time on subagent coordination, attack timings, and continuous observation. The title of leader in such a tournament does not yet mean the ability to play: all participants remained at the novice level. The checklist of failures from the matches — decision timings, subagent coordination, continuous observation — is applicable to auditing your own agent workflows, because the same failure modes occur in any environment where the world changes faster than the model can think.

What is still unknown / limitations

The table cannot yet be considered a reliable measure of model strength: each configuration has only 18 games, so gaps like 100% vs 88.9% are statistically indistinguishable from noise, and the data has no repeated series, seeds, and map pool description. There is also no measured human baseline — agents played only against each other. There is also an attribution problem: Grok 4.6 configurations ran in their own harness, and there is no factorial comparison of "one model in different harnesses" and "one harness with different models" in the data, so it does not follow from one tournament that Grok 4.6's failure is "product" and "fixable in the harness," nor does the opposite follow. The non-monotonic influence of the reasoning level can be either noise from a small sample or a real trade-off between the length of reflections and the real-time deadline — this is a key question to check on a larger sample. Finally, this is a research project by one enthusiast, not a product, and it has no immediate commercial effect.

Sources

Author

Look at AI, editorial team