Google announced a new robots.txt report within Google Search Console and at the same time said it will sunset the old robots.txt tester tool. The new tool shows which robots.txt files Google found for the top 20 hosts on your site, the last time they were crawled, and any warnings or errors encountered.
Google also said on X, “We’re also making relevant info available from the Page indexing report. As part of this update, we’re sunsetting the robots.txt tester.”
To access this, go to the settings link for your Search Console profile or click here. This is what it looks like:
Here is what the report looks like for this site, yes, I have a critical error. Here are those details and note, you can request recrawl of your robots.txt file:
This help document goes through a lot more about this report.
Some are not happy that the robots.txt tester tool is gone but Bing still has one:
Good alternatives to Google's depreciated tool to validate your robots.txt:
• Merkle: https://t.co/33yMW9IlPB
• Ryte: https://t.co/jYAGuHN2q5
• Screaming Frog (in-crawl): https://t.co/zsr2KMiNmC— Cyrus SEO (@CyrusShepard) November 15, 2023
Good alternatives to Google's depreciated tool to validate your robots.txt:
• Merkle: https://t.co/33yMW9IlPB
• Ryte: https://t.co/jYAGuHN2q5
• Screaming Frog (in-crawl): https://t.co/zsr2KMiNmC
I love the new robots.txt reporting (it’s in settings btw), but removing the robots.txt Tester? I love that tool and it can be super-helpful. There are other tools that do similar things, including one from Bing, but I’d love to see that stay in GSC. 🙂 https://t.co/hMIJQjfpQV
— Glenn Gabe (@glenngabe) November 15, 2023
I love the new robots.txt reporting (it’s in settings btw), but removing the robots.txt Tester? I love that tool and it can be super-helpful. There are other tools that do similar things, including one from Bing, but I’d love to see that stay in GSC. 🙂 https://t.co/hMIJQjfpQV
Forum discussion at X.
The content at the Search Engine Roundtable are the sole opinion of the authors and in no way reflect views of RustyBrick ®, Inc
Copyright © 1994-2023 RustyBrick ®, Inc. Web Development All Rights Reserved.
This work by Search Engine Roundtable is licensed under a Creative Commons Attribution 3.0 United States License. Creative Commons License and YouTube videos under YouTube’s ToS.
