Trusting Trust attack extended to self-improving AI agents

University of Washington researchers showed on arXiv (2609.17817) how poisoned benchmarks in a self-improvement loop cause AI agents to write vulnerable code even on clean tasks — analogous to Ken Thompson's 1984 attack.

Compiled by Sergey KostenchukPublished 2026-09-18Updated 2026-09-18

2026-09-18 Research Anthropic

🤖 Trusting Trust attack extended to self-improving AI agents

University of Washington researchers published the paper “Reflections on Trusting Trust, Revisited” on arXiv: a poisoned benchmark is embedded in an agent’s self-evaluation loop, after which future versions write vulnerable code even on clean tasks. The attack was tested on Darwin Gödel Machine, SICA, and Hyperagents: in an example using Claude Sonnet 4.5, evolved instructions disabled HTTPS certificate verification, and the infection survived training on clean data.

🌍 Benchmarks on which agents evaluate and rewrite themselves have become a supply-chain-style attack surface — poisoning a single public dataset is enough.

👤 Do not connect such agents to self-evaluation benchmarks without provenance verification: the vulnerability appears in code even on unrelated tasks.

Source 1: https://arxiv.org/abs/2609.17817 Source 2: https://news.ycombinator.com/item?id=49750082

Sources