Google's John Mueller says AI training crawlers usually don't offer sitemap submission and suggests default sitemap names or RSS feeds to help them find sites.

Google’s John Mueller says AI training crawlers usually don’t give site owners a way to submit a sitemap, and he suggests a default file name or an RSS feed for sites that want those crawlers to find their content.

He made the comments with Martin Splitt in the Oct. 1 episode of Google’s Search Off the Record podcast, titled “Do sitemaps still matter?” He also said he has seen AI crawlers access his sitemap and RSS files in his own server logs.

Mueller said site owners who want to keep a sitemap private can give it an unusual file name, leave it out of robots.txt, and submit it to Google directly. The downside is that other systems can’t find it, he said, and Bing would probably need its own submission.

AI training crawlers are different because “they usually don’t have any kind of a Console or any setup where you can submit a sitemap file.” For sites that want their content in AI systems, he suggested “either you stick to the generic naming, call it sitemap.xml, or you focus on RSS feeds.”

The sitemaps protocol says a robots.txt Sitemap line is independent of the user-agent line, so it isn’t tied to any one crawler’s rules. Feeds are easier to find, he noted, because they’re usually linked from a page’s HTML head.

“I’ve seen that happen in my server logs where some AI crawler accesses my sitemap file,” he said, and he’s seen the same with his RSS files.

Source link