New spam guidance says to noindex or block crawling on pages made for sharing. Google's docs say a URL blocked in robots.txt can be indexed.
Google has recently created an additional “Only index content meant for Search” section within the spam guide it updated on Oct. 26. The new section advises websites with pages made for direct sharing, collaboration or utility to set a noindex tag or use a robots.txt directive to block crawlers by default for those pages.
According to Google’s changelog this is an addition relating to “blocking indexing of link-shared content.”
The new section includes a link to the introductory page about robots.txt, and according to that introductory page the robots.txt file “is not a mechanism for keeping a web page out of Google.”
Google’s new “Only index content meant for Search” section follows the guide’s advice on spotting spammy accounts and reads:
“If your service lets users create shareable pages, generated outputs, or interactive tools, consider whether that content is meant to be found through search engines or only accessed directly by people with a link.”
“When pages are created for direct sharing, collaboration, or utility rather than public search discovery, apply the noindex rule (using a robots meta tag or X-Robots-Tag HTTP response header) or block crawling in robots.txt by default.”
” The updated sentence adds “tools that let users publish shared pages, generated content, or web apps,” and says spammers may create a large number of “pages or sites” where it used to say sites.
Source link
