Google explains the steps to take to ensure a CDN migration won't backfire and cause a negative impact to SEO.
Googlers Gary Illyes and Cherry Prommawin explain how moving to a CDN can unexpectedly increase crawling and overwhelm the origin server, despite eventually improving website performance. They also share that attempts to control crawler traffic can sometimes accidentally block Googlebot and harm Search visibility.
Illyes and Prommawin explained that moving to a CDN can cause Googlebot to increase crawling because faster server responses signal that the site’s infrastructure can handle more requests. Google’s automated crawling systems interpret the improved performance as an indication that more crawling is possible and responds by gradually increasing the crawl rate.
CDN caches are initially empty and the CDN only starts caching when a browser or bot requests a page at which point the request has to be forwarded to the origin server, where the actual website is at.
When Google increases the crawl rate, the initial crawl is handled by the origin server. But now Googlebot is hitting the server more aggressively and it’s this that can overwhelm the origin server, initially.
“For one, suddenly the crawling metrics suggests the site can handle a larger amount of crawling. And on the flip side, because the CDN caches aren’t yet populated, the origin server still has to serve the requested pages.
Basically, CDNs take the valve off the hose, so now crawlers are drinking from a fire hose.”
Source link







