SEO bots unable to crawl the website after connecting it to Cloudflare

Hi,

I have connected my website to Cloudflare’s CDN. However, while doing the site SEO, I came to know that the SEO crawlers are unable to crawl my site and index it. It seems that Cloudflare is somehow restricting the bots from crawling the site. What I get is an Error:400 “the server cannot or will not process the request”. I don’t understand the root cause of this; might be some security restrictions etc.

Any suggestions or solutions would be appreciated. Thanks already.

The Firewall Events in the Cloudflare Dashboard are your best friend.

https://dash.cloudflare.com/?to=/:account/:zone/firewall

2 Likes

Thanks for your response Erictung :blush: Do you suggest I should turn off all the firewall rules or should I exclude the crawling bots from the firewall rules? I don’t exactly know.

Please post a screenshot of the Firewall events that you are seeing.

I have the following firewall rules configured for the client:

Is there any way I can let the crawlers bypass Cloudflare and crawl the site for indexing purposes?

Do you mean on some usual, like Ahrefs, Moz, SEMRush or some other?

I believe you could bypass them by a part of their user-agent string - at least - or by their IP addresses, etc.

  • (http.user_agent contains "dotbot") or (http.user_agent contains "rogerbot") or (http.user_agent contains "SemrushBot") or (http.user_agent contains "Ahrefs") or (http.user_agent contains "RavenCrawler") or (http.user_agent contains "Screaming") or (http.user_agent contains "seolyt")

Therefore, I am really not sure why to use all of them to check SEO score as they seems to work similar, and why to index your Website on all of them? One is enough, at least, or not? (maybe some comparasion, but …)
For my point of a view, Google is far enough for most,
Why give others (and public) the information/inputs of your Website SEO score? :slight_smile:
But, that’s again, just my opinion.
Or, just use some offline tools like SEOPowerSuite, etc.

1 Like

I would create a new firewall rule which allows “Known Bots” and move it to the top so that this firewall rule gets executed first.

2 Likes

Thank you for the user-agent string and your suggestions regarding these crawlers.

Thanks, Erictung. I will surely add a new firewall rule to whitelist known bots and check the crawling process again.

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.