Block AI Scrapers and Crawlers block legit GoogleBots?
Full
Enable Block AI Scrapers and Crawlers
Visit Security > Events
IP’s blocked
66.249.68.38
66.249.68.32
ASN Google
Hostname: crawl googlebot
It was a short test. May have blocked more if kept enabled.
Screenshot:
The user agent containing GoogleOther shows that its not a GoogleBot as such it is “GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.”
I am not an expert, but see that the Block AI Scrapers and Crawlers is blocking IPs like:
66.249.77.1
66.249.77.2
66.249.77.3
66.249.77.5
66.249.77.6
66.249.77.4
66.249.77.7
66.249.77.8
Are these not googlebot IPs?
For instance 66.249.77.7 is classed as ’ an IP address from within our benign crawler allowlist. We confidently believe it is not a bad bot’ by AbuseIP
https://www.abuseipdb.com/check/66.249.77.7
As you can see from the User Agent: “GoogleOther” it is a GoogleIP but in that instance it is not being used by the standard GoogleBot so should be blocked
Reading about it, Google say that ‘GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.’
It has ‘one-off’ crawled my site 39,000 times in the last 8 hours since I turned on the Block AI Scrapers and Crawlers feature on, which seems excessive
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.
