Google's John Mueller clarified whether site owners can use robots.txt to block Googlebot from downloading specific resources that users see.

Google’s John Mueller answered a question about whether it’s okay to block Googlebot from indexing an age-verification popup that website visitors are required to see. His answer offers some clarity about whether it’s okay to purposely hide some kinds of content from Googlebot and explains how site owners can do it.

Google requires publishers to allow Googlebot to crawl everything that a user might see but that wasn’t always the case. There was a time in the early days of SEO where SEOs blocked Google from downloading JavaScript and CSS files in order to optimize the crawl budget and encourage better website indexing.

But there came a time when Google began rendering web pages and subsequently required site owners to allow Googlebot to download all resources that a user would see. Some sites that continued to block Google from downloading critical resources temporarily lost their rankings.

So the question about whether it’s okay to block Google from indexing a popup is reasonable.

The SEO asked on Reddit whether it’s okay to use robots.txt to prevent Googlebot from indexing an age-verification popup that a website is legally required to display to visitors.

The person described a hypothetical website that loads an age-verification popup from a separate domain through an API. The question was whether blocking Googlebot from crawling that external domain would prevent the popup from being rendered.

com – for legal reasons it has an age verification pop up that sits on a separate domain and is inserted on the page via an API.

Source link