Bing cannot access my site when cloudflare on

i have contact Cloudflare customer support as i am on the paid version and they are being less than helpful, which makes me wonder why i am paying for the paid version.

When i have Cloudflare on i cannot do a site scan to find out problems with my site. When i request a site scan it only does one pgae but when i turn off Cloudflare it then scans the whole site.

I have been turned down by bing news because they cannot access the site.

can anyone please let me know if they have this problem and how to solve it

Greetings,

Thank you for asking.

Despite now you are using Cloudflare for your domain name, I guess?, may I ask did you tried to get Bing News before you started using Cloudflare for your domain? :thinking:

I am asking because of the strict guidelines … similar to Google News has got.

Furthermore, maybe you have to “allow” and/or “whitelist” Bingbot by its user-agent, or by adding the Bing/Microsoft AS number to Firewall Rules or IP Access Rules?

  • there might be some caveats as far as there are also fake “Bingbog” user-agents out there, even some strange attack patterns from the Microsoft AS number …

Do you see any of the requests (events) being challenged or “blocked” by navigating to the Firewall tab → Overview at Cloudflare dashboard for your domain name? :thinking:

Related to the Firewall Rules, have you got some configured already, like country blocking, etc.?

There is an option even to try out and use cf.client_bot which includes “good” and “verified” bots:

yes u was accepted for bing news previously but because they cannot access my site with Cloudflare on i am no longer on bing news. but more importantly is the site scan

how do i add bing to Cloudflare

Hi,

I just had a similar issue - they key is to set your firewall overview page tp 30 mins and filter by the MS ASN ( 1. AS8075 - MICROSOFT-CORP-MSN-AS-BLOCK) then initiate a scan from the big webmaster site and watch to see if you get any blocks in the firewall.

For me the issue was the Cloudflare specials rulset was blocking bingbot. As soon as i turned that off everything was ok. Pretty stupid the protection for a fake bing bot false positives the real bing bot. I am new to Cloudflare and this is the first thing that left me less than impressed.

I did not have to create a rule to enable bingbot via the firewall after i disabled this rule set, though a firewall rule to allow through the ASN works too.

This was the json report that made me realized the issue. (see the meta data section where it says the ruleset) good luck.

{
  "action": "block",
  "clientASNDescription": "MICROSOFT-CORP-MSN-AS-BLOCK",
  "clientAsn": "8075",
  "clientCountryName": "US",
  "clientIP": "13.66.205.190",
  "clientRequestHTTPHost": "www.mydomain.com",
  "clientRequestHTTPMethodName": "GET",
  "clientRequestHTTPProtocol": "HTTP/1.1",
  "clientRequestPath": "/robots.txt",
  "clientRequestQuery": "",
  "datetime": "2022-02-20T08:31:00Z",
  "rayName": "6e06604a79816810",
  "ruleId": "100202",
  "rulesetId": "",
  "source": "waf",
  "userAgent": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)",
  "matchIndex": 0,
  "metadata": [
    {
      "key": "group",
      "value": "cloudflare_specials"
    },
    {
      "key": "rule_message",
      "value": "Anomaly:Header:User-Agent - Fake Bing or MSN Bot"
    }
  ],
  "sampleInterval": 1
}

Using Bingbot verification tool, the IP 13.66.205.190 comes from the Microsoft AS number and contains Bingbot in the user-agent, but unfortunately you are wrong as this one is a real fake one and Cloudflare blocked it correctly and reasonably (as a WAF rule specified Anomaly:Header:User-Agent - Fake Bing or MSN Bot"):

Your free choice :wink:

1 Like

I assure you it was really bing - i initiated a scan from the big webmaster toolset to generate that.
tl;dr that set of rules really does break genuine bing activity that i initiated.

I guess MS bingbot verification tool cannot be trusted, bloody awesome.,

for anyone searching and hitting this thread this was the bing functionality being used Bing Webmaster Tools

If it is, then MS is at fault for not respecting their ruleset to help third parties verify their bot. I would report this to Microsoft as it can cause similar confusion in the future.

1 Like

yes just did, would certainly help explain some of the arguments i have seen in this community where people swear blind CF blocked legit bing traffic and others vociferously said it couldn’t have They are both right. Maybe folks can learn something from that…

thanks for pointing me in right direction, i will be monkey on MS’s back on this one

2 Likes

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.