sel logo
Search Engine Land » SEO » Googlebot crawl rate tool in Search Console is going away
Google is deprecating the crawl rate limiter legacy tool within Google Search Console on January 8, 2024. Google said this feature is no longer useful because Google said it has improved its “crawling logic and other tools available to publishers.”
What is the crawl rate limiter. It is a tool within the legacy version of Google Search Console that lets you communicate to Google crawl your site less than it currently does. Google has historically recommended against limiting the crawl rate unless you are seeing server load problems that are definitely caused by Googlebot hitting your server too hard.
You can access the tool over here, until it is removed. Here is what it looks like:
Why Google is removing it. Gary Illyes from Google said, “with the improvements we’ve made to our crawling logic and other tools available to publishers, its usefulness has dissipated.”
“Googlebot reacts to how the site–or more specifically the server handling the site– responds to Googlebot’s HTTP requests. For example, if the server persistently returns HTTP 500 status codes for a range of URLs, Googlebot will automatically, and almost immediately slow down crawling. Similarly, Googlebot slows down automatically if the response time for requests gets significantly longer. If you do experience unusually heavy crawling that your site can’t manage on its own, refer to this help article,” Gary Illyes added.
Illyes explained that the rate limiter tool within Google Search Console “had a much slower effect.” He said the tool often would “have taken over a day for the new limits to be applied on crawling.”
He added that the tool was “rarely” used by site owners and those who used it “in many cases set the crawling speed to the bare minimum.”
Crawl rate change. Google said with the deprecation of the crawl limiter tool, Google is setting the minimum crawling speed to a lower rate, comparable to the old crawl rate limits. So Google will “effectively continue honoring the settings that some site owners have set in the past if the Search interest is low, and our crawlers don’t waste the site’s bandwidth,” Illyes added.
What if you have issues. If you have issues with crawling, Google said you can read this help document and use the report form to let Google know.
Why we care. If you have been using this crawl rate tool, keep in mind it will be going away. So set a notification on your calendar to see what, if any, impact this has on your server when the feature is turned on.
Related stories
New on Search Engine Land
About the author
Related topics
Get the daily newsletter search marketers rely on.
See terms.
Learn actionable search marketing tactics that can help you drive more traffic, leads, and revenue.
March 15-16, 2023: SMX Munich
Online Aug. 16-17: SMX Master Classes
Start training now: SMX Next
Start training now: SMX Advanced
Discover time-saving technologies and actionable tactics that can help you overcome crucial marketing challenges.
April 15-17, 2020: San Jose
Upgrade Your 2024 B2B-Marketing Game With This Powerful Ad Channel
4 B2B Commerce Trends to Catapult You Ahead of the Competition
Hype Check: What Marketers Really Want to Know About Generative AI
Enterprise Marketing Attribution and Performance Management Platforms: A Marketer’s Guide
Enterprise Account-Based Marketing Platforms: A Marketer’s Guide
Enterprise Marketing Work Management Platforms: A Marketer’s Guide
8 New Winning SEO Tests
Meet your new AI-powered marketing assistant!
Get the must-read newsletter for search marketers.
Topics
Our events
About
Follow us
© 2023 Third Door Media, Inc. All rights reserved.
Third Door Media, Inc. is a publisher and marketing solutions provider incorporated in Delaware, USA, with an address 88 Schoolhouse Road, PO Box 3103, Edgartown, MA 02539. Third Door Media operates business-to-business media properties and produces events. It is the publisher of Search Engine Land the leading Search Engine Optimization digital publication.

source