Gary Illyes and Cherry Prommawin from Google elucidate how transitioning to a Content Delivery Network (CDN) can inadvertently escalate crawling activity, thereby placing undue stress on the origin server, despite the eventual enhancement in website performance.
They further elaborate that efforts to regulate crawler influx can occasionally result in inadvertently blocking Googlebot, subsequently sabotaging Search visibility.
Implications of Transitioning to a CDN on Googlebot Crawling
Illyes and Prommawin articulated that migration to a CDN can provoke an uptick in Googlebot’s crawling activity.
This phenomenon occurs primarily because swifter server responses convey to Google’s automated systems that the site’s infrastructure is capable of managing additional requests.
The enhancement in performance prompts Google’s crawling apparatus to interpret this as an opportunity for increased crawling and, gradually, to amplify the crawl rate.
Initially, CDN caches are devoid of content; they commence caching only upon a page request from a browser or bot. At that moment, the request is relayed to the origin server, which hosts the actual website content.
As Google escalates the crawl rate, the initial crawling is borne by the origin server. However, the resultant scenario sees Googlebot engaging with the server more vigorously, potentially overwhelming it in the process.
Gary Illyes articulates:
“Initially, the crawling metrics imply that the site can support a greater volume of crawling activity. Conversely, since the CDN caches remain unpopulated, the origin server continues to fulfil the requested pages.
Essentially, CDNs release the floodgates, allowing crawlers to access an unrestrained flow of requests.”
Cherry Prommawin added further insights:
“Under normal conditions, our crawlers function according to a pivotal principle. When a server responds more rapidly, the crawlers deduce that the infrastructure is sound and gradually enhance the crawl velocity.
…Thus, the crawlers observe the swift responses, become exceedingly enthusiastic, and commence crawling the entire site at an exponentially increasing pace. Yet, this presents a pitfall.
Many mistakenly believe that a CDN can effectively manage a surge in crawl requests from the outset; however, …The cache requires time to fill with assets from the site.”
Illyes further explained that for the first request for any given page, the CDN must retrieve the content from the origin server to adequately populate its cache.
It is only upon the second request that the cache is sufficiently filled, allowing it to fulfill its role.
Illyes continued:
“This means your origin server might endure substantial pressure from an influx of requests. To offer a fair assessment, our crawlers do take time to ramp up; it’s not an instantaneous occurrence.
An unfilled cache signifies that your origin server continues to bear the brunt of the demand. Until the cache is adequately populated, your CDN is not yet functioning as a protective barrier.”
Key Takeaway:
It is crucial to note that a CDN cannot deliver an uncached URL without first retrieving it from the origin server, a process that only commences after a request for that specific URL is made.
Given that crawlers consistently visit numerous URLs, they can generate substantial traffic directed at the origin server prior to the complete population of the CDN’s cache.
The Issue of CDN Bot Protection and Inadvertent Googlebot Blocking
The subsequent concern revolves around the accidental blocking of Googlebot, which can result in diminished search visibility as Google is unable to crawl and index the URLs of a website.
Illyes and Prommawin delineate this issue in the context of a website administrator inadvertently activating a setting on Cloudflare, which results in Googlebot being obstructed.
This predicament is quite tangible; there was a notable incident involving a well-known website whose considerable traffic vanished upon the activation of Cloudflare, only to return once it was deactivated.
The process of diagnosing this situation proved arduous, with customer support facing challenges in identifying the root cause.
It is plausible that, as noted by Illyes and Prommawin, the administrator had unknowingly enacted a stringent bot filter that blocked Googlebot’s access.
Illyes exemplified an administrator’s potential error:
“Upon witnessing an influx of traffic, they presume they’re being aggressively targeted by botnets and, understandably, they become alarmed.”
What frequently transpires next is that an administrator may activate “blanket security protocols” that inadvertently inhibit Googlebot’s access.
Prommawin elucidated further:
“In a state of panic, one may access their CDN dashboard and enable settings to completely block bots.
…Should the CDN discern automated requests, it intercepts them with a CAPTCHA challenge, demanding visual confirmation of human engagement by clicking on images of traffic lights.
…Invariably, automated requests are flagged as bots, with subsequent interception resulting in the CAPTCHA barrier. As our crawlers are incapable of identifying a traffic light, they consequently fail the verification process.”
Prolonged blocking of Googlebot can ultimately lead to the exclusion of pages from Google’s index.
Prommawin recommended that website administrators configure their CDNs to permit access for known and trusted crawlers.
This strategy, referred to as whitelisting, ensures that only authorized crawlers are granted access, while restricting all others.
Additionally, the implementation of CDN-level rate limiting for Googlebot can be advantageous when requests to the origin server threaten to induce downtime, negatively impacting genuine visitor traffic.
Recommendations for Pre-Warming CDN Cache
Upon initially adopting a CDN, the cache remains devoid of a website’s content. It only becomes populated following requests from either browsers or crawlers.
The technique of cache pre-warming entails proactively requesting content from a site to ensure its availability on the CDN.
This preemptive action ensures that the content is already accessible via the CDN prior to user requests, facilitating swift delivery without necessitating retrieval from the origin server.
Illyes advocates for “pre-warming” the cache through systematic retrieval of URLs to efficiently populate the cache.
“To effectively augment your site with a CDN, consider pre-warming the caches by programmatically retrieving your most crucial URLs through the CDN before allowing an influx of search engines.”
Concluding Insights
The predominant message from Google underscores that transitioning to a CDN has the potential to create an unforeseen scenario characterized by accelerated crawling coupled with escalated demands on the origin server.
This dynamic can inadvertently precipitate the establishment of stringent security protocols that may obstruct Googlebot’s attempts to access the website.
Key considerations for SEOs and site proprietors include:
- Utilizing a CDN could instigate an increase in Googlebot’s crawling activity due to faster server responses, even when the CDN’s cache remains unpopulated, leading to heightened requests to the origin server.
- The migration to a CDN may inadvertently result in various pages vanishing from Google Search due to stringent security protocols aimed at thwarting malicious bots, which can simultaneously exclude Google’s crawlers.
- Google recommends pre-warming caches, permitting trusted crawlers, and implementing temporary controls on crawling in circumstances where server demand may become excessive.
For further insights, peruse Google’s updated CDN documentation here.
Access the Google Podcast:
Source link: Searchenginejournal.com.





