AI giants back unified opt-out for AI scraping in robots.txt

🛡 AI giants back unified opt-out for AI scraping in robots.txt

OpenAI, Google, Microsoft, and Anthropic will support the "policy token" — a unified industry opt-out signal proposed by Kevin Kelley (Anthropic) through the IETF. Anthropic has already implemented it, OpenAI will do so in the coming months, and Google and Microsoft are expected to follow in 2027.

🌍 Publishers no longer need a separate opt-out for each provider — a single entry will suffice for multiple. This is a step toward harmonization with the EU AI Act, where robots.txt is recognized as the primary way to express preferences.

👤 Websites will be able to use a single entry in robots.txt to prohibit model training on their content, with a separate parameter for grounding (use as a source for answers). Meta supported the idea but has not yet confirmed it will comply with the signal.

Source 1: https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out Source 2: https://openai.com/index/helping-websites-express-ai-training-preferences/