We got an email from Google Merchant Centre saying to ensure we don’t block Google Bot.
We are not sure how to ‘not’ block Google Bot.
So in short we need to allow “Storebot-Google" and other Google crawlers to access our site.
It is very hard to find a simple answer on this topic in the community forums. Many topics are for advanced and technical users.
Is this not something that Cloudflare would automatically allow to over ride for its users in case they accidentally block Google bots? For example, we have Bot Fight Mode on. Would this block Google too???
Also, where do we specifically allow Google to crawl our site in Cloudflare? Is it under Rules > Configuration rules?
And would this work?
So we’d select “User Agent” and “Contains” and the value would be “Storebot-Google”?
Or is there another way to do this more effectively?
Thank you for your help, to anyone who can help.
Generally speaking @priti, if a user-agent isn’t explicitly blocked, then it is allowed. You should not need to create a rule to explicitly allow the bot.
Yes but if we have bot fight mode on, wouldn’t that potentially explicitly block the Google Bot?
Per Cloudflare Bot documentation
If you have a smaller domain and have identified a bot problem, we recommend Bot Fight Mode…
So do you have a problem with bots already, or are you activating Bot Fight Mode as a just in case measure?
The documentation also says
Cloudflare bot solutions identify and mitigate automated traffic to protect your domain from bad bots.
So (as I read it) it specifically targets bad bots. Because as stated here
Cloudflare maintains an internal list of Verified Bots that are associated with search engine optimization (SEO), website monitoring, and more.
which again, I would take to mean they aren’t going to block bots that are legitimate/non-bad.
I might expect they wouldn’t block Google’s bots. But honestly, I cannot say for certain one way or the other.
Thank you this is helpful. Do you know if there is a way to know for sure if they don’t block google’s bots? I’ve seen other people try to set this up through the rules and what not, but it’s just so complicated how they do it.
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.