The standard way to evaluate model switching inside LLM agents — replaying recorded trajectories over stitched logs — assesses states that simply do not occur in live runs. This is the conclusion of the paper “The Replay Gap” (Ashritha Gonuguntla, COLM 2026, arXiv:2608.08239), in which live SWE-bench agent trajectories were branched at checkpoints and continued with a different model. For the LLM router market, whose economics depend on such replay benchmarks, this is a direct hit to the methodology; for engineers, it is a signal to re-verify model cascades in their own agent pipelines on live rollouts rather than on logs.

image
image

What happened

In August 2026, the paper “The Replay Gap” (Ashritha Gonuguntla, arXiv:2608.08239) was presented at the Efficient Reasoning Workshop as part of the COLM 2026 conference. Instead of replay evaluation, which replays a recorded trajectory and swaps the model at selected points, the author branched live SWE-bench agent trajectories at checkpoints, rebuilt the environment for each branch, and continued it with a different model — about 900 rollouts in total across six paired runs. After a model swap, 61 to 94 percent of subsequent actions are rewritten, and only 3 percent of replayed states remain valid. The log-stitching evaluator failed on every decisive success call, and the similarity between its predicted patches and the actual ones ranged from 0.00 to 0.11. All five outcome reversals recorded in the work — a model upgrade saved an unsolved instance, a downgrade lost the only found solution — occurred only in branches with switching; there were none in the 359 control branches. A separate finding was that FP8 quantization makes temperature-0 mode non-deterministic: identical models on FP8 inference diverge at more than 90 percent of branch points, whereas on AWQ inference there are almost no divergences.

Context

Replay evaluation emerged because live agent rollouts are expensive: to avoid restarting the agent after every model swap, benchmarks replay a recorded trajectory and assume that the rest of it will not change. The methodological weakness of this assumption is that an agent trajectory is a chain of states where each action changes the environment: swapping one call moves the run into a different branch, along which the recorded log trail no longer proceeds, so the replay evaluator validates scenarios that do not happen in reality. Within this logic, the product category of per-step routing grew — choosing the cheapest sufficient model not only per request but at every step of the agent. The side result about FP8 has broader significance: since inference quantization breaks temperature-0 determinism, the classic requirement that “identical input plus temperature-0 equals identical output” is not met by default in modern serving stacks.

Why this matters for the industry

The market bet on agent routing rests on replay metrics that, according to these data, are systematically invalid for agents: the promise of “the cheapest sufficient model at every step” is evaluated on stitched logs, and routing decisions made on them do not transfer to live runs. For LLM router vendors, this is the risk that claimed savings will not survive the first live audit by a customer. For startups, a window opens in “trusted” agent evaluation, where the product becomes the methodology itself — branching rollouts with environment rebuilding and control branches, not another model. The FP8 finding additionally hits static model comparisons as such: without fixing the inference stack, such comparisons lose reproducibility, and “reproducible inference” may over time become a requirement for vendors. If independent checks replicate the numbers in other domains, eval tools are expected to start adding branching rollout modes, and the customer question about metric provenance will become a standard due diligence item.

Why this matters for users

If you are testing cascades or routing between models in your own agent pipeline, metrics calculated on logs without a live environment restart cannot be considered validated for your system. An alternative is already available: the author’s harness and all trajectories are publicly released on GitHub and Hugging Face, so your stack can be run through branching rollouts with environment rebuilding and control branches, and then the live rollout can be compared with replay log readings. Until such a check, claims of the form “a cheap model performs no worse” inside agents should reasonably be considered unconfirmed. Separately, it is worth checking whether your serving runs on FP8: if so, temperature-0 cannot be considered reproducible, and for model comparisons it makes sense to fix the inference stack and consider alternative quantization, for example AWQ. For a buyer, a simple check of whether the metrics claimed by a vendor were calculated on live rollouts is already a mandatory checklist item.

What is still unknown / limitations

The entire evidence base was collected on SWE-bench agents in six paired runs with about 900 rollouts, so transferring the conclusion to other classes of agents, such as browser or tool-use agents, and to “the overall market bet on routers” is a hypothesis about impact, not a result of the work. The market interest signal is still early: three points and zero comments on Hacker News is an academic signal, not a market revaluation. Expectations about the emergence of branching modes in eval tools and a revision of vendor benchmarks also remain interpretations until independent replications.

Sources

Author

Look at AI, editorial team