Engineer Carlos Tapia built the open sandbox-conformance test bench, which runs public sandbox tests for AI coding agents — sandbox-runtime, nono, Codex CLI, landrun, and others — against each other using a unified set of contracts. In the October 15, 2026 run on Linux 6.8, 61 of the 79 recorded escapes, or about 77%, occurred not because a configured rule failed, but because the necessary rule was entirely absent from the default configuration. The bench is published as an open repository, so any of its results can be independently verified.

image
image

What happened

Carlos Tapia published the sandbox-conformance repository along with the results of a run conducted on October 15, 2026, on Linux 6.8. By October 26, 2026, the corpus had grown to version v1.1 and includes 196 contracts compiled from the public test suites of the projects themselves: sandbox-runtime, nono, Codex, Deno, bubblewrap, and OpenHands. Each contract formalizes a specific attack, the legitimate control that must stop it, and the verifiable effect on disk or network. The most frequent category of escape in this series was trust-handoff: of the 79 recorded cases, 37 describe a scenario in which an agent writes a malicious git hook to the project's working folder, which then executes outside the sandbox when a human runs a standard git commit. As of October 27, 2026, the author has sent six reports to vendors via security programs: four to sandbox-runtime and two to Codex, two of which remain unresolved.

Context

Sandboxes for AI coding agents isolate the execution of generated code using a set of rules that define what is allowed to be written to disk, what network access is permitted, and what execution is allowed. This implies that their protection relies not only on the quality of the engine applying the rules, but also on the completeness of the configuration itself. The approach chosen by Tapia brings the methodology of compatibility suites, long used to test languages and runtimes, into agent environment security: there is a unified corpus of expectations, and each implementation is run against it under equal conditions. Crucially, the corpus is compiled not from attacks devised by a single researcher, but from the sandboxes' own test suites: vendors' "native" expectations are presented to one another, which structurally reduces the risk of tailoring results to a desired conclusion. The outcome of such a run is falsifiable, and the corpus is versioned, so comparing sandboxes across versions remains reproducible.

Why this matters for the industry

For the industry, the bench turns the claim "we have a sandbox" from marketing into a verifiable contract: a unified corpus of contracts provides a benchmark by which sandbox-runtime, nono, Codex CLI, landrun, and any other implementations can be compared under equal conditions, and the completeness of the default configuration becomes a measurable product parameter. The submitted reports give vendors not abstract criticism, but a specific list of holes to close through standard security programs. Teams building agent environments can already run the corpus in CI as a mandatory eval-gate when selecting or updating a sandbox and as a basis for security scoring of such environments. If vendors close the reports and integrate the corpus into release regression, hardened defaults and an increased number of adapters are likely, and coverage of the public corpus may eventually become a de facto compatibility level of the kind "sandbox conforms to v1.x" — this part is an interpretation, not an established fact.

Why this matters for users

If you run AI agents like Codex or Claude Code on your own machine, the practical takeaway is one: you should check whether your sandbox covers trust-handoff scenarios. An agent may leave a file in .git/hooks or modify an editor config in the project's working folder, and these files will later be executed by legitimate tools outside the sandbox, within your normal workflow. This also includes secret masking: without an explicit rule, the environment may allow token output to leak out. Verification is free: the sandbox-conformance repository is open under the Apache-2.0 license, contains comparative result tables by version and raw evidence, and a custom sandbox adapter can be run against the same contracts used to test the sandboxes mentioned in the article.

What is still unknown / limitations

The conclusions rely on a single run performed on October 15, 2026, on Linux 6.8: the figure of about 77% indicates a deficit in default configurations in this specific environment and on this date, but says nothing about the ceiling of enforcement quality, so the conclusion that enforcement has become a commodity does not follow from this run's data. The metric also does not measure how high-quality the enforcement engines are where rules are configured. Two of the six reports sent to vendors remain unresolved, so part of the results is not formally confirmed by the manufacturers. The corpus is currently a single-author project, and its further growth is not guaranteed. The public signal is early: at the time of writing, the Hacker News discussion had 1 point and 0 comments, so broad community verification has not yet occurred.

Sources

Author

Look at AI, editorial team