sel logo
Search Engine Land » SEO » Googlebot crawl rate tool is now gone
As expected, Google removed the crawl rate setting tool within Google Search Console. Last November, Google told us they would deprecate the crawl rate limiter legacy tool within Google Search Console on January 8, 2024.
January 8th came and went and so did the crawl rate limiter with it.
Google said this feature is no longer useful because Google said it has improved its “crawling logic and other tools available to publishers.”
It’s gone. Now when you try to access the tool for a specific property within Google Search Console, you are redirected to the notice of the tool going offline.
What was the crawl rate limiter. It is a tool within the legacy version of Google Search Console that lets you communicate to Google crawl your site less than it currently does. Google has historically recommended against limiting the crawl rate unless you are seeing server load problems that are definitely caused by Googlebot hitting your server too hard.
This is what the tool looked like:
Why Google removed the tool. Gary Illyes from Google said, “with the improvements we’ve made to our crawling logic and other tools available to publishers, its usefulness has dissipated.”
“Googlebot reacts to how the site–or more specifically the server handling the site– responds to Googlebot’s HTTP requests. For example, if the server persistently returns HTTP 500 status codes for a range of URLs, Googlebot will automatically, and almost immediately slow down crawling. Similarly, Googlebot slows down automatically if the response time for requests gets significantly longer. If you do experience unusually heavy crawling that your site can’t manage on its own, refer to this help article,” Gary Illyes added.
Illyes explained that the rate limiter tool within Google Search Console “had a much slower effect.” He said the tool often would “have taken over a day for the new limits to be applied on crawling.”
He added that the tool was “rarely” used by site owners and those who used it “in many cases set the crawling speed to the bare minimum.”
Crawl rate change. Google said with the deprecation of the crawl limiter tool, Google is setting the minimum crawling speed to a lower rate, comparable to the old crawl rate limits. So Google will “effectively continue honoring the settings that some site owners have set in the past if the Search interest is low, and our crawlers don’t waste the site’s bandwidth,” Illyes added.
So hopefully you won’t need to use this tool anyway.
What if you have issues. If you have issues with crawling, Google said you can read this help document and use the report form to let Google know.
Why we care. If you have been using this crawl rate tool, keep in mind it no longer works and there is no way to access the tool anymore. You may want to check how this change impacts your crawl rate going forward, over the coming days and weeks.
Related stories
New on Search Engine Land
About the author
Related topics
Get the daily newsletter search marketers rely on.
See terms.
Learn actionable search marketing tactics that can help you drive more traffic, leads, and revenue.
Online Feb. 28-29: SMX Master Classes
Online June 11-12: SMX Advanced
Online Nov. 13-14: SMX Next
Discover time-saving technologies and actionable tactics that can help you overcome crucial marketing challenges.
April 15-17, 2020: San Jose
Decode the Future of AI and Privacy with Metadata Mastery
Maximizing Marketing ROI With a Composable Stack: Separating Reality from Fallacy
Upgrade Your 2024 B2B-Marketing Game With This Powerful Ad Channel
Customer Data Platforms: A Marketer’s Guide
Marketing Automation Platforms: A Marketer’s Guide
Enterprise Marketing Attribution and Performance Management Platforms: A Marketer’s Guide
Demystifying Inbox Deliverability Changes Coming in 2024
Meet your new AI-powered marketing assistant!
Get the must-read newsletter for search marketers.
Topics
Our events
About
Follow us
© 2024 Third Door Media, Inc. All rights reserved.
Third Door Media, Inc. is a publisher and marketing solutions provider incorporated in Delaware, USA, with an address 88 Schoolhouse Road, PO Box 3103, Edgartown, MA 02539. Third Door Media operates business-to-business media properties and produces events. It is the publisher of Search Engine Land the leading Search Engine Optimization digital publication.

source