OpenAI, Google, Microsoft and Anthropic back IETF 'policy token' in robots.txt to opt out of AI scraping
OpenAI, Google, Microsoft and Anthropic said they will adopt a proposed IETF 'policy token' placed in a site's robots.txt that lets publishers express AI training preferences across multiple crawlers. Anthropic has already implemented the signal; Google and Microsoft plan support next year and OpenAI expects implementation in the coming months; Microsoft will offer more granular options and sites will still appear in Bing search results.