Microsoft Bing Crawler Blocked

Hi several People had the same Problem like me also that their Cloudflare hosted Websites
had blocked IP Adresses from Microsoft from which the WebCrawler tryed to access the Webpages.

I just checked the email of my Website and got a Notification from Microsoft Bing that their Webcrawler could not access my Cloudflare Website.
See snipet here it say Problems with Crawling of the Website:

Return-Path: [email protected]
Received: from smtpi.msn.com (co1gmehub06.msn.com [65.55.52.234])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits))
(No client certificate requested)
Message-ID: b4f623af6cd34618bfe7578f6e2efe0b-JFBVALKQOJXWILKNK4YVA7CCNFXGOV3FM[email protected]microsoft.com
MIME-Version: 1.0
From: Bing Webmaster Tools [email protected]
Sehr geehrter Webmaster,

Dies ist ein täglicher Bericht zu offenen Problemen, die in Zusammenhang
mit Ihrer Website aufgetreten sind.

Fehlerhafte Crawls1

Can somebody from Cloudflare reach out to the Microsoft Bing Search Engine Team be it over email or
Twitter and inform them abotu this Problem ?
Looks like this are valid IP Adresses used by the Bing Crawler and should be not Blocked.

I deactivated temporary the Security Firewall Rule that checks the Bots if they are valid and do not get anymore this Problem with blocked crawlers for the moment.

Thanks in advance.

Continuing the discussion from New web site has 5.11k blocks in 24 hours:

Why Wait
Don’t wait for an answer, find it fast! Search for #CommunityTip error:
Example: #CommunityTip 521

Test Before You Post
Unsure of the issue? Test before posting using the Cloudflare Diagnostic Center: https://www.cloudflare.com/diagnostic-center/

1 Like

The link you referenced showed an apparent impostor as that address did not identify as Microsoft crawler.

Your case on its own might be a bit different, as you actually got a notification from Microsoft, but here it will be important to know which IP address it is, as your firewall settings might simply block that.

Go through your list of firewall events and identify the IP address and why it was blocked.

Hi sandro.

Hmm that is strange.
I did a whois for one of his IP Adresses and it resolvs to Microsoft Incorporation.
Here is a whois for his first ip address he posted:
It did not resolv back as writen in the docs becouse they at Microsoft guess maked a mistake but the IP Adresses from where the Crawler request the Pages are legit.

52.250.106.194
NetRange: 52.224.0.0 - 52.255.255.255
CIDR: 52.224.0.0/11
NetName: MSFT
NetHandle: NET-52-224-0-0-1
Parent: NET52 (NET-52-0-0-0-0)
NetType: Direct Assignment
OriginAS:
Organization: Microsoft Corporation (MSFT)
RegDate: 2015-11-24
Updated: 2015-11-24
Ref: https://rdap.arin.net/registry/ip/52.224.0.0

OrgName: Microsoft Corporation
OrgId: MSFT
Address: One Microsoft Way
City: Redmond
StateProv: WA
PostalCode: 98052
Country: US
RegDate: 1998-07-10
Updated: 2017-01-28
Comment: To report suspected security issues specific to traffic emanating from Microsoft online services, including the distribution of malicious content or other illicit or illegal material through a Microsoft online service, please submit reports to:
Comment: * https://cert.microsoft.com.
Comment:
Comment: For SPAM and other abuse issues, such as Microsoft Accounts, please contact:
Comment: * [email protected].
Comment:
Comment: To report security vulnerabilities in Microsoft products and services, please contact:
Comment: * [email protected].
Comment:
Comment: For legal and law enforcement-related requests, please contact:
Comment: * [email protected]
Comment:
Comment: For routing, peering or DNS issues, please
Comment: contact:
Comment: * [email protected]
Ref: https://rdap.arin.net/registry/entity/MSFT

OrgAbuseHandle: MAC74-ARIN
OrgAbuseName: Microsoft Abuse Contact
OrgAbusePhone: +1-425-882-8080
OrgAbuseEmail: [email protected]
OrgAbuseRef: https://rdap.arin.net/registry/entity/MAC74-ARIN

OrgTechHandle: MRPD-ARIN
OrgTechName: Microsoft Routing, Peering, and DNS
OrgTechPhone: +1-425-882-8080
OrgTechEmail: [email protected]
OrgTechRef: https://rdap.arin.net/registry/entity/MRPD-ARIN

Here is a Screen Shot of my Firewall Blocked Request in the last 24 Hours filtered by the BingBot Problem:

As you see nearly 200 Request alone in the last 24 Hours were blocked by the Firewall for the IP Address which belongs to Microsoft.

whois 51.143.124.155

inetnum: 51.140.0.0 - 51.145.255.255
org: ORG-MA42-RIPE
netname: MICROSOFT
descr: Microsoft Limited UK
country: GB
admin-c: DH5439-RIPE
tech-c: MRPA3-RIPE
status: LEGACY
mnt-by: RIPE-NCC-LEGACY-MNT
mnt-by: MICROSOFT-MAINT
created: 2015-05-21T17:21:54Z
last-modified: 2016-07-25T09:38:58Z
source: RIPE

organisation: ORG-MA42-RIPE
org-name: Microsoft Limited
org-type: LIR
descr: Microsoft Corporation AS8075
descr: To report suspected security issues specific to
descr: traffic emanating from Microsoft online services,
descr: including the distribution of malicious content
descr: or other illicit or illegal material through a
descr: Microsoft online service, please submit reports
descr: to:
descr: * https://cert.microsoft.com
descr: For SPAM and other abuse issues, such as Microsoft
descr: Accounts, please contact:
descr: * [email protected]
descr: To report security vulnerabilities in Microsoft
descr: products and services, please contact:
descr: * [email protected]
descr: For legal and law enforcement-related requests,
descr: please contact:
descr: * [email protected]
descr: For routing, peering or DNS issues, please
descr: contact:
descr: * [email protected]
address: One Microsoft Way
address: WA 98052
address: Redmond
address: UNITED STATES
phone: +1 425 882 8080
fax-no: +1 425 936 7329
abuse-c: MAC274-RIPE
mnt-ref: RIPE-NCC-HM-MNT
mnt-ref: MICROSOFT-MAINT
mnt-by: RIPE-NCC-HM-MNT
mnt-by: MICROSOFT-MAINT
created: 2004-04-17T12:18:10Z
last-modified: 2020-07-01T16:22:30Z
source: RIPE # Filtered

role: Microsoft Routing, Peering, and DNS
address: One Microsoft Way
address: Redmond, WA 98052
nic-hdl: MRPA3-RIPE
mnt-by: MICROSOFT-MAINT
created: 2014-08-26T16:25:24Z
last-modified: 2014-08-26T16:25:24Z
source: RIPE # Filtered

person: Divya Quamara
address: One Microsoft Way
address: Redmond, WA 98052
phone: +1-425-882-8080
nic-hdl: DH5439-RIPE
mnt-by: MICROSOFT-MAINT
created: 2014-08-26T16:24:14Z
last-modified: 2016-02-19T07:09:41Z
source: RIPE

% Information related to ‘51.140.0.0/14AS8075’

route: 51.140.0.0/14
origin: AS8075
descr: Microsoft
mnt-by: MICROSOFT-MAINT
created: 2020-07-22T15:05:24Z
last-modified: 2020-07-22T15:05:24Z
source: RIPE # Filtered

This looks like a old Problem that a lot of People faced.
I solved the Problem Temporary following this How To here.
For the Moment i dont have this Firewall Blocking anymore but People
from cloudflare should really reach out to Microsoft and solve this Problem.
It costed me several Ranking Positions in Bing and DuckDuck go allready.

The provided Answer this Request do not resolve back to what the Docs say could be wrong and outdated.
it could be also very well be that this is some kind od Developers testing the newest Version of the Crawler before deploying it.

I think the most plausible answer to this Problem is that this is actually the Microsoft Advertising Team craweling the Website. I have a registered account with this email at Microsoft Advertising becouse i wanted do advertising at DuckDuckGo and they do crawl website to check if everything is good.

I would clearly not leave this situation like now.
Will do more diging why this happened especially the last 7 Days such much.
Before it was not that much Intense.
Guess Bing is maybe creating a New Index and doing a Core Index Update.
Then it would be very very bad …

Both addresses belong to Microsoft but are not necessarily part of their Bing infrastructure.

Neither address identifies as official address, as mentioned in the link you referenced. If they really are official addresses Microsoft would need to identify them correctly.

I addressed that before.

I agree with you that this are IP adresses that Belongs to Microsoft and it looks like they are from different parts of the Corporation aka guess Advertising which in turn has configured the Crawler not right and becouse of this Cloudflare is now blocking the Request.
Cant tell for sure what the Problem is.
Clear is that me like others got a lot of this Request from Microsoft IP Adresses Blocked by the Firewall and now i even got a email from the bing webmaster tools that the crawler experienced crawler problems the last days.

I am not saying Cloudflare is doing something wrong.
In contrary i do think Cloudflare is doing everythign right.
The Problem is clearly on the Bing Crawler.

Either way something need to be done by cloudflare as this can not be let like this i think.
its bad for them too when a lot of Webmaster get Email from Bing telling them theri Cloudflare Website are not reachable …

Not sure what they should do. If Microsoft deployed a rogue crawler they need to fix this on their side. Cloudflare can only adhere to the official identification mechanism.

You best contact Microsoft and clarify with them if this actually their crawler and if it is, let them know to identify it correctly.

You can always open a support ticket.

I do clearly think that this ugly Problem was very bad handled by the Community and Cloudflare itslef.

You have People writing here since now over 10 Months that in Google Merchant Center and else where they have Errors Reprted related to crawling the Cloudflare Hosted Webpages.

and then as a Solution its provided that this are Fake Bots ?
How can this be Fake Bots and imposters when the failed request appear in the official Google Merchant Center and Bing is evern writing emails to the Webmasters they should check their Website.

If this are all Fake Bots request not one of this request will anywhere be reported by Google Merchant Center as failed request officialy but they are reported so this are not Fake Bots Request but clearly wrong configured Official Crawler Bots that are Blocked by Cloudflare which should not happen however !

The community does not really need to handle anything in the first place :wink:

I already addressed that twice. The majority of them will be false reports.

Valid point, but that seems to be only you so far. No need to inflate the whole thing.

Again, you need to take this up with Microsoft. It is them who are wrongly identifying their own software.

I got this message days after enabling CloudFlare Specials.

After disabling CloudFlare WAF rule #100202, Bing can access the site again.

I think the problem is that Bingbot is using IP addresses that fail in their own “Verify Bingbot” test. So, you can’t blame CloudFlare, but I wish there was a way to use rule #100202 without blocking legitimate Bing traffic.

1 Like

I have been noticing this issue too (was just to start report this here until i see this topic), and indeed is Cloudflare that is blocking legitime BingBot Ips.

I have received many Bing Webmaster alerts in the past few days alerting that Bing bots are not able to crawl my website, getting a 403 error (error below is in Portuguese, but mostly says that bing detected pages with the 403 error, the page that appears when a WAF rule is activated)

After check this, i notice that Cloudflare Firewall was indeed blocking many ASN Ips from Microsoft (AS8075), with the WAF Rule " 100202 - Anomaly:Header:User-Agent - Fake Bing or MSN Bot".
The Ips being blocked, checking with the Bing Webmaster Tools (that have a verify IP from BingBot) indeed appear as not valid BingBot network. But the fact is that the errors are building up because those Ips are being block.

Those Ips have the correct useragent from bing: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
But this can indeed be faked. Neverless, doesn’t appear that way.

For instance, one IP that i have found being blocked is “51.143.124.155”. Under the Bing Webmaster Tools, this IP doesn’t appears to be a Bing Bot.

BUT is one of the Ips used by Bing to crawl.

Since then, i have added the Microsoft ASN to bypass any waf rule, and the errors in Bing Webmaster Tools started to decrease. This proves that there is indeed something going on with Cloudflare+Bing search bots.

I don’t know if Microsoft changed the way how BingBot works, or started use a different ASN for that, but the fact is that Cloudflare is blocking legitim Bing Ips from accessing the website.

Also, this is not the first time that we are seing this error. A quick search in the Cloudflare Comunity shows many instances of the same problem, under different users.
https://community.cloudflare.com/search?q=bing%20bot

Now, this probably isn’t Cloudflare fault, maybe is Bing that changed the way how it works. But something is going on and hurting the SEO from websites in Bing.

2 Likes

That is certainly an option, though this will also allow other crawlers through.

Bottom line, you need to talk to Microsoft and they need to fix their configuration. Additionally, as I already mentioned, you can certainly open a support ticket with Cloudflare too. But overall it is Microsoft who needs to fix this.

1 Like

Indeed, allowing the ASN from Microsoft is not the best… but is better that having the website hurt in Bing SEO because of Cloudflare.

In the end, is Cloudflare that is blocking the requests - and they are legit requests made by Microsoft.
Bing can have changed something, but is Cloudflare fault that the SEO is being impacted because of this.

Not really. If you configure a rule which should block crawlers which are not Microsoft’s and Microsoft runs such a rogue crawler as well, then the block is working as intended.

As I said numerous times, Microsoft needs to fix their identification.

The “rule” is default from Cloudflare WAF Protection - not a custom one.
And yes, Microsoft should fix this, and Cloudflare also, so they have responsability to update the WAF rules that provide.

I didn’t say it was a custom one. The issue is Microsoft is not declaring the crawler, but then I did say that quite a few times already :wink:

Yes, not deny in that. :slight_smile:
The point here is that is not only Microsoft. Cloudflare have responsibility in maintain and update the WAF rules as well…

And that’s what Cloudflare is doing based on Microsoft’s declaration. If Microsoft does not handle that properly, there is little that Cloudflare can do.

1 Like

Its pretty clear that both Google and Microsoft Cloudflare Firewall Rules are sabotaging the access to Cloudflare Hosted WebSites !

Reason for this is that neither Google nor Microsoft are actually keeping up with their recommendation that their Crawlers resolves back to the right domain back.

So every Google and Microsoft Departement in every different Country uses different configured Bots that are not respecting its own set recommendation which in turn makes Cloudflare sabotaging the Access to the Website becouse it holds on to the Recommendation.

I also would like to see this two rules changed so it only blocks bots that are not comming from the IP Range of the Companys.
As long the Requests are done by legit Company IP addresses the requests should be allowed by the Firewall.

1 Like

That would include their whole hosting services, which would render the rules pointless as everyone hosting with them could pose as their crawler.