A shared robots.txt line could let sites opt out of AI training with several providers at once, but companies involved are handling it differently.

Google, Microsoft, and OpenAI announced plans on Friday to honor a shared robots.txt line for AI training preferences. A site could use it to state one training preference to every provider that takes part, instead of writing rules for each company’s crawler.

The line is from an Internet Draft that Anthropic’s Kevin Kelley submitted to the AI Preferences working group at the IETF (same date). This is listed in the IETF’s document tracker as an individual draft, which does not represent official IETF standardization activity. In a post to the working group’s email list, Kelley indicated that Anthropic has started development to provide this feature.

Based on their provided timelines, none of the three companies currently supports the line. Google and Microsoft described how each would handle it differently. According to Google, if a site sets up a shared opt-out line, Google plans to treat it like a Google-Extended block when a website doesn’t specify Google-Extended, and this control also applies to grounding within Gemini Apps. Microsoft wants sites that use its NoArchive and NoCache tags to switch to new preferences by March 1, 2027.

txt. An ordinary rule group, that a site might write under what the draft calls a policy token, __usage_ai-training, would be written like it always has been, using Allow and Disallow. The draft provides an example of how one might opt out of AI training on the entire site, except for the /press-kit/ folder.

Source link