POLICY · REGULATION · #2849
OpenAI, Google, Microsoft and Anthropic back IETF 'policy token' in robots.txt to opt out of AI scraping
OpenAI, Google, Microsoft and Anthropic said they will adopt a proposed IETF 'policy token' placed in a site's robots.txt that lets publishers express AI training preferences across multiple crawlers. Anthropic has already implemented the signal; Google and Microsoft plan support next year and OpenAI expects implementation in the coming months; Microsoft will offer more granular options and sites will still appear in Bing search results.
KEY POINTS
- OpenAI, Google, Microsoft and Anthropic said they will adopt a proposed IETF 'policy token' placed in a site's robots.txt that lets publishers express AI training preferences across multiple crawlers.
- Anthropic has already implemented the signal; Google and Microsoft plan support next year and OpenAI expects implementation in the coming months; Microsoft will offer more granular options and sites will still appear in Bing search results.
- A shared robots.txt token standardizes and simplifies publisher control over AI crawling and training, potentially limiting unauthorized scraping and changing how models access web data.
WHY IT MATTERS
A shared robots.txt token standardizes and simplifies publisher control over AI crawling and training, potentially limiting unauthorized scraping and changing how models access web data.