Cloudflare blocking bingbot

i have read lots of posts and no one seems to be answering this question. I want to know how to allow the bingwebmaster tools bot to crawl my site. at the moment when you go into bingwebmaster tools and ask for a site scan, it will only scan one page as Cloudflare is blocking it.

I would like to know if anyone finally has the answer to solve this problem. i have asked Cloudflare customer support and as others have said they have been no help

I am also wondering if this is also stopping google from properly crawling the site.

when i turn off Cloudflare bingwebmaster scan can properly crawl my site but as soon as i activate it, it then stops again

I am sorry to hear this.

I really don’t know why this happens to some users, and cannot replicate it → I am afraid it must have to be something with their security settings at Cloudflare.

I’d suggest you to rather submit the sitemap xml file → maybe too much requests from Bingbot coming and Cloudflare is blocking them? :thinking:

Can you share some information from you with us here so we could try to troubleshoot this further?:

  1. When you navigate to Firewall tab → Overview, and see some request being “blocked” or “challenged” which is from Bingbot, which service is being triggered?
  2. Can you share the screenshot of it?
  3. What’s the IP of the request which is being blocked?
  4. What is your Security Level set to (None, Low, Medium, High, I am under an attack)?
  5. Bot Fight Mode is enabled or disabled?
  6. Are you using some of the custom Firewall Rules?
  7. How about Page Rules?
  8. Are you using Free or a Paid plan?

Using Bingbot verification tool you can check if it’s a real Bingbot or not, despite of coming from the Microsoft AS number:

Bing is on the “good” and “known” list:

What kind of error do you experience?

I doubt. Google is also on a known list of “good bots”, despite there are again, the “bad” requests from the Google AS number from which Googlebot has got official IPs published too:

Possible solutions:

  1. Creating a Firewall Rule with cf.client.bot “true” and action “allow”, make sure it is the 1st Firewall Rule from above.
  2. Creating a Firewall Rule with user-agent contains bing and action “allow”, make sure it is the 1st Firewall Rule from above.
  3. Risky one but … add Microsoft AS number (AS8075) to “allow” in IP Access Rules.

Otherwise, kindly I’d suggest you to write a ticket to Cloudflare support due to your account and/or domain issue and share the ticket number here with us so we could escalate this issue:

  • Login to Cloudflare and then contact Cloudflare Support by clicking on the Get More Help button. If you get automatic reply, reply and indicate to it you need more help and reference to this topic
  • Or send an an e-mail to support[at]cloudflare[dot]com from your e-mail associated with your Cloudflare account
1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.