Google's new robots.txt video says the standard covers crawling, not how content gets used, and recommends percent-encoding non-English robots.txt paths.

Robots.txt, as the internet standard RFC 9309 defines it, governs which URLs a crawler may request, not what happens to the content afterward, Google’s Gary Illyes said in a Search Central video published Wednesday. To keep one AI crawler out while allowing others, he said, you have to target that crawler’s user-agent token in the file.

The video is the second episode of The Non-Canonical Cabinet, where Illyes and Cherry Prommawin of Google Search Relations take edge-case questions from the community. Illyes is also co-author, with Mozilla’s Martin Thomson, of an IETF Internet-Draft that would give robots.txt a standard way to state usage preferences. Prommawin recommended writing non-English rule paths in percent-encoded form for crawlers other than Google’s.

The episode’s question was whether robots.txt can allow AI crawling of a website and keep that same information out of generative model training. According to Illyes, “So as defined by RFC 9309, robots.txt is an access preference protocol.” He went on to say robots.txt “doesn’t dictate what a system does with the data after it has already crawled it.” If you want to completely block a specific AI crawler from accessing your entire website, Illyes stated, you need to “target their specific user agent tokens explicitly in the file.”

, No-AI-Chicken-Training) which was shown on-screen and would be ignored by a parser. txt specification page states that the four elements supported are User-Agent, Allow, Disallow, and Sitemap; invalid lines will be ignored by Google.

Source link