Jonathan Hendler released MediationBench, an open benchmark where language models play both sides in 62 scripted disputes while the tested model acts as a mediator. After each message, the mediator decides whether to speak, wait, or end the conversation, and a score from 0 to 100 is assigned by the judge, GPT-5.6 Sol. In a controlled study, a "skilled" mediation strategy outperformed simple echo-reflection by +13.17 points in a public context; however, the benchmark currently holds only provisional status: the strategy effect does not transfer stably across backends, and the leaderboard has not yet undergone inter-judge validation.

image

What Happened

Jonathan Hendler published the MediationBench benchmark on mediationbench.com and presented it in a post titled "Show HN: MediationBench – testing AI mediators in 62 simulated conflicts" on Hacker News. In the benchmark, language models play both sides in 62 scripted disputes—ranging from business partnership breakdowns to driveway and credit conflicts—while the tested model acts as the mediator. After each message, the mediator chooses one of three actions: speak, wait, or end the conversation, with the dialogue fitting within a window of 3 to 25 rounds. The final score from 0 to 100 is assigned by the judge, GPT-5.6 Sol, based on five weighted components: cooperation (25%), depth of resolution (25%), revealed hidden interests (20%), balance of obligations (15%), and mediator quality (15%). In a controlled study where Qwen3-235B served as the mediator and the Stheno model played the parties, the "skilled mediation" strategy outperformed the control strategy of "reflective listening" by +13.17 points (90% CI 9.67–17.55) in a public context and by +7.27 (90% CI 1.31–10.52) with private briefs. On the current v3.0 leaderboard, where Gemma 4 31B fixes the parties, the top spots are held by Claude Opus 5 with a score of 71.2, Claude Opus 5.5 with a score of 70.3, and Claude Fable 5.1 with a score of 69.9. Along with the benchmark, a working draft of the paper "MediationBench: Auditing AI Mediation in Synthetic Conflict" (dated September 13, 2026) was published, and anyone can run their own model in the role of mediator.

Context

For LLM benchmarks, this is a non-standard format: the protocol evaluates not the content of replies, but the timing of control decisions—that is, the choice of action at each turn—which is closer to evaluating agentic behavior than to question-answering tests. The control isolation is structured so that the parties are always fixed by a separate model—Stheno in the controlled study and Gemma 4 31B in the v3.0 leaderboard—thereby testing only the mediator and reducing contamination between the model and the environment. The methodological apparatus includes preregistered rules, a fixed set of disputes, and a holdout of 38 conflicts, while the controlled study provides confidence intervals, which remains rare for projects of this genre. Before MediationBench, the industry lacked an open protocol allowing models to be compared as mediators, so the publication of a fixed set of conflicts with a primary judge becomes a baseline for a new vertical.

Why This Matters for the Industry

The industry now has its first public protocol for "auditing" AI mediation: a fixed set of disputes, a primary judge, and preregistered rules provide developers and regulators with a measurable way to compare models as mediators before they enter real HR, client, and neighborly disputes. The main engineering signal from the study is that the interaction of "strategy × model" is measured, not a universal mediation ability: the full effect was obtained on Qwen3-235B, partial reproduction on Kimi K2.5, and no effect on gpt-oss-120b, so declaring the leaderboard leader the possessor of an abstract mediation capability would be incorrect. Three elements are reusable from the protocol: turn-level gating ("speak, wait, or end"), a five-component rubric for dialogue quality, and a simulation A/B test of strategies before deployment; the combination of "fixed scenarios, holdout, preregistered rubric, and provisional labeling" serves as a ready-made template for mature evaluation design for internal evals. Teams can check negotiation agents against current leaders for free without spending on their own evals; if the provisional status is removed after connecting three or more judges, the leaderboard could become a working reference for selecting models for conflict workflows, and scenario-based evals of "soft" behavioral skills could eventually stand alongside coding and agentic benchmarks in vendor checklists.

Why This Matters for Users

Anyone can open the leaderboard to compare which models—Claude Opus 5, Kimi K3, GLM 5.3, GPT-5.6 Sol, and others—best resolve conflicts, read the working draft of the paper and a dialogue example, and run their own model in the role of mediator for free. For the average reader, this is currently a reference point rather than a service: the author emphasizes that all parties in the disputes are AI, so the benchmark measures model behavior in simulation, not safety, fairness, or the durability of agreements with real people. The practical takeaway is to use the leaderboard as a starting point when selecting a model for tasks involving conflict dialogue, but not as a guarantee that the same model will perform identically in a live dispute.

What Is Still Unknown / Limitations

Final comparisons currently hold provisional status: the score is assigned by a single primary judge, GPT-5.6 Sol, there is no inter-judge validation, and the status is removed only when evaluated by three or more judges, so the sensitivity of scoring to judge choice remains unverified. The proof of the strategic layer's value is conditional: it was obtained on one backend, in the scenario with private briefs, the lower bound of the 90% CI (1.31) is close to zero, and the community has not yet confirmed the transferability of the effect to other models. The paper exists as a working draft dated September 13, 2026, and has not undergone external discussion—the Hacker News post had collected 1 point and 0 comments at the time of writing. Unknowns include judge consistency among themselves, results of checking the 38 conflicts from the holdout, and the transfer of results to live participants; the benchmark does not measure the safety, fairness, or durability of real agreements.

Sources

Author

Look at AI, editorial team