Cloudflare has introduced a new AI traffic management system that replaces simple blocking with detailed bot classification, dividing them into search engines, agents, and model training tools.
What Happened
Cloudflare has implemented a system that classifies crawlers into three types: Search (indexing for search engines), Agent (performing actions on behalf of a user in real-time), and Training (collecting data for model training). Starting September 15, 2026, on sites using advertising, bots in the Training and Agent categories will be blocked by default. Meanwhile, Search bots will remain accessible to preserve referral traffic. To manage these rules, the BotBase catalog has become available to enterprise customers.
Context
Previously, AI traffic management was limited to a binary model: either full blocking or full access. New mechanisms, such as extending robots.txt via Content Signals, aim to create a transparent ecosystem of interaction between websites and AI agents, allowing content owners to protect intellectual property without losing visibility in search engines.
Why It Matters for the Industry
This decision changes the rules of engagement between content owners and AI developers, turning intellectual property protection into a technically feasible function. The implementation of such standards could lead to the formation of a mature 'data economy' market, where access to content for model training becomes a regulated or paid service at the network infrastructure level.
Why It Matters for Users
Website owners gain a powerful tool to protect their content from unauthorized use in training datasets while maintaining SEO performance. Users of AI agents may encounter changes in resource access rules, which will require developers to more carefully configure identifiers and headers to pass through the new filters.
What Is Not Yet Known / Limitations
Some experts point to potential negative consequences for the AI development ecosystem, calling such measures a direct blow to data collection methods.
Sources
Author
Look at AI, Editorial Team