Model-Switching Replay Tests in LLM Agents Measure the Wrong Thing

The study “The Replay Gap” (COLM 2026) showed that evaluating model switching in LLM agents using replayed trajectories is invalid: after swapping the model, 61–94% of subsequent actions are rewritten, and only 3% of replayed states remain valid.

Compiled by Sergey KostenchukPublished 2026-08-31Updated 2026-09-18

2026-08-31 Research HuggingFace

🤖 Model-Switching Replay Tests in LLM Agents Measure the Wrong Thing

The study “The Replay Gap” (COLM 2026) branched live SWE-bench agent trajectories (~900 rollouts) and continued each with a different model. After the swap, 61–94% of the next actions were rewritten, only 3% of replayed states were valid, and all 5 outcome reversals happened only during switching.

🌍 The market bet on LLM routers relies on replay benchmarks that are invalid for agents: they assume the trajectory after a model swap does not change. In addition, FP8 inference makes temperature-0 non-deterministic, breaking static model comparisons.

👤 If you are testing model cascades in agentic pipelines, do not trust metrics from logs without a live environment restart. You need branching rollouts; the author's code and trajectories are open, and the methodology can be reproduced on your own stack.

Source 1: https://arxiv.org/abs/2608.08239 Source 2: https://github.com/AshrithaG/replay-gap

Sources