Google DeepMind has conducted the first double-blind evaluation of a proprietary frontier model: Gemini Flash Lite was tested by four external organizations — Singapore's AI Safety Institute, OpenMined, AVERI, and MLCommons — using confidential benchmarks, with evaluators not seeing the model weights and Google not seeing the test prompts. For the first time, independent verification of a frontier system was carried out without mutual disclosure of secrets, and the industry has received a working template for such an audit.

image
image

What Happened

The evaluations were conducted in a GPU enclave based on Google Cloud Confidential Space with NVIDIA H100 accelerators and Intel TDX, hardware encryption of RAM and VRAM (AES-256 / AES-XTS-256), remote attestation according to the IETF RATS standard, and reproducible builds; orchestration was performed via PySyft, and the syft-restrict policy limited evaluators to an allow-list of permitted methods. The overhead for confidential computing was less than 5%, and the enclave is destroyed after the run without residual state. The methodology and results of the pilot are described in the technical report DeepMind Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing.

Context

The motivation for the pilot is measurable, not hypothetical. According to the study by Xu et al., 2024, cited in the report, signs of test data leakage were detected in approximately half of the 31 models checked, meaning that a significant part of the benchmarks is known in advance to the evaluated systems. This picture is exacerbated by benchmark hacking: the report mentions a case where a laboratory ran 27 private model variants on Chatbot Arena and published only the best result. Previous independent audits of proprietary models were stuck in a confidentiality dilemma: the developer had to either accept the evaluators' confidential prompts on their own infrastructure or hand over the model weights to a third-party organization, and for frontier models, a scheme in which both parties kept their secrets did not exist. Confidential computing technologies were already available, but before this pilot, they had not been applied to live external evaluations of proprietary models.

Why This Matters for the Industry

For the AI industry, double-blind evaluation for the first time provides a cryptographically verifiable mechanism for independent auditing of frontier models, eliminating the dilemma of "who will trust whom first": the provider no longer needs to disclose weights and intellectual property, and the evaluator — confidential benchmarks. This is a direct response to benchmark hacking and contamination, which undermine trust in leaderboards, and a sign of the emergence of a new infrastructure layer of verifiable AI evaluations, in which trust becomes computable. The protocol has already been tested with the national AI Safety Institute and MLCommons, so it can be considered as a template for regulatory checks, including sensitive areas such as cybersecurity and the government sector. Engineering teams can already study the published stack — Google Cloud Confidential Space, IETF RATS attestation, PySyft with syft-restrict — and prototype similar workflows for their own tasks, such as auditing RAG pipelines.

Why This Matters for Users

The main practical effect for the reader is the ability to understand the primary source: the DeepMind technical report is a rare case where confidential computing, including Confidential Space, H100 enclaves, remote attestation, and PySyft with a method allow-list, is applied not on paper, but in a live pilot with external organizations. From this also comes a useful correction to the usual reading of leaderboards: ratings may be based on benchmarks known in advance to the models, so rating figures should be treated with caution. It is important to understand the built-in limitation of the scheme: it requires trust in the cloud provider and hardware vendor, so double-blind evaluation is a step towards independent evaluations, not a full guarantee of them.

What Is Still Unknown / Limitations

The pilot is still a single case: this is n=1 on one lightweight model Gemini Flash Lite, and the transfer of the protocol to larger frontier models, other types of eval workloads, and other enclaves has not been demonstrated. The sources do not contain data on a publicly described independent repetition of the pilot by other teams, nor is there an API, pricing, or latency data, meaning that a self-service path for ordinary production pipelines is absent, and the protocol has not yet been integrated into standard evaluation practice. Open questions remain as to whether the protocol will become part of standard evaluation practice and whether it will extend to ordinary public benchmarks, not just confidential audits.

Sources

Author

Look at AI, editorial team