BingBot is being blocked by Cloudflare

I have setup a free Cloudflare account in the summer for my site and everything I thought was working great.

I have since found my site has disappeared from Bing. I logged into Bing Webmaster Tools and found that Bingbot could not fetch any pages, robots.txt or sitemap files from my site. Other bots like GoogleBot can.

I did contact Bing Support and this is what they said:

During my investigation I was able to identify that we are in fact having issues reaching your site URL with our crawlers.

When checked the crawl status of the your site URL with help of our internal tool, receiving crawl error as " OtherFailure" response, instead should receive 200 response.

I can simulate BingBot using their user agent and have access to the website so it appears Cloudflare is blocking BingBot by IP. I am not sure of this as I see no notifications in the Cloudflare Dashboard about blocking BingBot.

Anyways I currently have paused Cloudflare from the Overview tab (lower right corner), and Bing is able to crawl my site.

What do I need to do to get Cloudflare to allow Bingbot to access my site?

Thanks

Cloudflare doesn’t block bingbot by default.

As this has come up many times before, please use the Search function and work your way through the suggestions already offered.

It was the first thing I did. I found more recent ones like:

Bingbot is blocked by cloudflare - Security - Cloudflare Community

Bingbot blocked with Error 4xx - Security - Cloudflare Community

(which you have responded to as well it looks like) but never seemed to find out what is causing the problem.

I did look at the Overview of the Firewall Events activity log and it says “No firewall events found matching your filters”. THis was over the last 24 hours which covers the time when I was doing URL inspections of my site in Bing Webmaster Tools and it was reporting Fetch errors (until at least I paused Cloudflare).

So I looked at more posts and didn’t find any with a real solution.

So how is Bingbot getting blocked. Since I am on the free version of Cloudflare I only have 3 rules setup for my WordPress site.

Challenge High Risk Traffic
No Direct Plugin Access
Block xmlrpc.php Attacks

The “Challenge High Risk Traffic” rule I had known bots set to threat level 14 for bots. I set it to 49 and still Bingbot was being blocked. I actually then disabled all rules and then ran a URL Inspect from Bing Webmaster Tools and still it was being blocked.

Looking at all 3 rules the “Activity Last 24 Hours” is showing nothing so it appears these rules are not being actually triggered by BingBot.

Where else can I look on the Cloudflare Dashboard to see where/how they are blocking Bingbot as nothing is displayed in the Firewall Rules activity log.

And Cloudflare is blocking bingbot for sure as if I pause Cloudflare, Bingbot can then fetch my pages.

Help!

I was hoping you would reply back. I have offered a lot more information and any of the suggestions offered when searching do not fit my situation. In fact most posts with issues with BingBot just stop responding with no solution.

I have disabled ALL my Firewall rules and Cloudflare still blocks Bingbot so in fact it does appear BingBot is blocked by default by Cloudflare.

If you have any other ideas I am willing to hear them…

Are you using any security plugins within your website i.e. within WordPress?
If you don’t have IP forwarding setup correctly then its possible your internal security is blocking authentic Bingbot from accessing the site.
Do you mind sending me your website address for me to run a few checks?

What tls version you using? Are you using minimum tls 3?

Tls is in the ssl section

Needs to be minimum 2 or lower, bing is like the only SEO bot that can’t work solely on tls 3 lol it stuffed me up once.

1 Like

I do have WordFence plugin installed (along with the Cloudflare plugin). My first thought it was a WordFence issue so I did deactivate the plugin with Cloudflare enabled and BingBot still was not able to access the site.

If you still want to run a few checks it is the website in my profile. I have currently enabled Cloudflare so you can run your checks.

When you perform a Bing Webmaster Tools Live URL check and it fails. Does it give you a response header (HTTP Response) by any chance that you can share here?

It is extremely unlikely Cloudflare would block authentic Bingbot traffic, my sites work fine with Bing so something isn’t configured correctly somewhere.

You don’t by any chance have any geo load balancing enabled on your site?
I also assume authentic Googlebot crawls the site fine?


Your website seems to be indexed in Bing?

Thanks JoshJ, GeorgeP and others who replied. JoshJ solution was the one. I had TLS set to 1.3. Setting it to 1.2 allowed BingBot to fetch my pages with CloudFlare enabled.

1 Like

Yeah iti s in Bing now because I have had paused Cloudflare for the last few days while I looked for a solution. Rather have my website in Bing than have Cloudflare enabled. BTW I just tested Josh solution and that was the issue. I had TLS set to 1.3 which BingBot cannot handle. Switching it to 1.2 fixed the problem.

haha, i knew it! happened to me once too to my frustration where all other bots worked fine. :stuck_out_tongue:

Don’t get me started on Bingbot I will end it here before I get banned from the forums :slight_smile:

good to hear it’s working.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.