OpenAI, Google, Microsoft, and Anthropic have announced support for a unified industry opt-out signal for AI scraping, embedded in the familiar robots.txt file. The mechanism, known as a policy token—a single preference token—was proposed through the IETF AIPREF working group and replaces the set of rules for each provider's crawlers with a single entry. Working support currently exists only at Anthropic; OpenAI promises implementation in the coming months, while Google and Microsoft plan to implement it in 2027.
What Happened
Four of the largest AI developers have publicly supported a unified machine-readable signal that allows websites to prohibit the use of their content for model training. The mechanism—a 'policy token,' or single preference token—was introduced by Kevin Kelley, Technical Lead for Standards at Anthropic, through the IETF AIPREF working group: the draft draft-kelley-aipref-training-preference-signal-00 'Machine-Readable Training Preference Signal' was updated on October 30, 2026. Of the four companies, working support currently exists only at Anthropic, whose crawlers are already required to respect the signal. OpenAI promises implementation 'in the coming months,' meaning it is currently a promise rather than an active mechanism, while Google and Microsoft plan implementation in 2027. Microsoft is additionally replacing NoArchive and NoCache tags with two independent parameters: AI Training, which regulates model training, and AI Grounding, which regulates the use of content as a source for generating answers. For this reform, Microsoft has named four dates: transitional settings open in Bing Webmaster Tools on December 15, 2026; the mechanism takes effect on February 1, 2027; the deadline for migrating sites from NoArchive and NoCache is March 1, 2027; and temporary grounding settings expire on September 30, 2027.
Context
Prior to this proposal, publishers wishing to block content from AI systems had to configure rules separately for each provider, listing crawlers one by one in User-Agent; the new signal reduces this set to a single token that works for multiple companies simultaneously. The move toward a common format is also driven by regulation: in the EU AI Act and the Code of Practice for GPAI providers, the robots.txt file is already recognized as the primary way to express website preferences, meaning the standard addresses not only convenience but also future regulatory requirements. The methodological basis of the reform is the separation of two fundamentally different operations in the ML pipeline: training changes model parameters, while grounding refers to accessing content as a citable source when forming an answer. The proposal itself is protocol and coordination work, where systemic value outweighs scientific novelty.
Why This Matters for the Industry
For the industry, the main benefit is the reduction of transaction costs in managing content rights: instead of a separate policy for each AI provider, a unified machine-readable layer and a cross-company control mechanism are emerging. A new direction of tools is arising around the token: parsers, validators, CMS plugins, migration services, and analytics, while laying the foundation for an AI licensing and compliance solutions market for businesses. Teams collecting data for training must make parsing and respecting the signal part of their pipeline and mark sources that are now closed—this is already relevant for Anthropic because its support is active. Startups have an unoccupied window: the map of who and how is complying with the signal has not yet been built, and creating it is itself a productive task. There is no direct effect on inference cost and latency—this is a data management mechanism, not model optimization.
Why This Matters for Users
If you run a website or blog, you can today use a single entry in robots.txt with User-Agent __usage_ai-training to prohibit the use of your content for training Anthropic models, and a separate parameter to limit grounding. Regular Bing and Google search is not affected by these settings, so visibility in search results is preserved. For most websites, there will be no practical changes until the end of the year: OpenAI support remains a promise, and Google and Microsoft will only connect in 2027. Owners using NoArchive and NoCache tags will need to migrate to Microsoft's new parameters—transitional settings in Bing Webmaster Tools are designed to provide time for this, and temporary grounding settings will be in effect for a limited period.
What Is Still Unknown / Limitations
An announcement of support is not the same as implementation, and implementation is not the same as verifiable behavior. Compliance with the signal during model training is not externally verifiable: there is no verification API or logs, and observation is possible only at the crawling level, for example through canary pages and monitoring fetch behavior. The standard does not cover the reuse of already collected corpora, derivative datasets, or distillation, so it only concerns future data collection. A telling detail is Meta, which supported the idea but did not confirm that it would technically comply with the token: the strength of the mechanism depends on the commitments of participants, not the protocol. It is unknown whether the standard will become a de facto norm for other providers, whether an independent market for compliance checks will emerge, and how companies that did not participate in the announcement will behave.
Sources
- The biggest names in AI are backing a new way to opt out of scraping — The Verge (Emma Roth)
- Making it easier for websites to express AI training preferences — OpenAI
- Supporting a shared AI training control for websites — Google Search Central Blog (John Mueller)
Author
Look at AI, editorial team