So in short we need to allow “Storebot-Google" and other Google crawlers to access our site.
It is very hard to find a simple answer on this topic in the community forums. Many topics are for advanced and technical users.
Is this not something that Cloudflare would automatically allow to over ride for its users in case they accidentally block Google bots? For example, we have Bot Fight Mode on. Would this block Google too???
Also, where do we specifically allow Google to crawl our site in Cloudflare? Is it under Rules > Configuration rules?
Generally speaking @priti, if a user-agent isn’t explicitly blocked, then it is allowed. You should not need to create a rule to explicitly allow the bot.
Thank you this is helpful. Do you know if there is a way to know for sure if they don’t block google’s bots? I’ve seen other people try to set this up through the rules and what not, but it’s just so complicated how they do it.