BingPreview/1.0b blocked by Bot Fight Mode

Security
#1

Hello,

Bot Fight Mode is blocking bingbot and BingPreview by default. I’ve managed to set working rules to allow bingbot, however no rule specified (no matter the order) will allow BingPreview to pass with the following:

User agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

IP Addresses (Range specified): 40.77.139.0 - 40.77.139.255 or 40.77.139.*

ASN: AS8075 MICROSOFT-CORP-MSN-AS-BLOCK

Note those ips where checked against Bing Webmaster “Verify Bingbot” Tool and they are shown as verified bots.

I’ve tried rules with User Agent “contains” or “equal”, IP Addresses “lists” and anything i can think of and all the requests are “JS Challenge” for those ips, no matter the order or priority of the specified rules.

Perhaps someone had or managed to fix this issue? Also it seems quite strange Cloudflare doesn’t recognize those ips by default, i assume those are new and maybe the database hasn’t been updated yet.

#2

Liaising internally on this. Will let you know if we require further action from you.

#3

Hi Luis,

What is the full IP address of the bingBot?

Also, what is the domain name and rayID of the blocked request in your firewall events? If you prefer, you can private message me this information.

Many requests, even from Microsoft ASNs may not be legitimate. Microsoft provides the below guidance to verify a bingBot:

https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26

https://www.bing.com/toolbox/verify-bingbot

https://www.bing.com/webmasters/help/Verify-Bingbot-2195837f

If we see that Bing shows that the bot is legitimate, I’ll be able to raise this issue with the Bot Management team.

#4

Hey there Cascading,

Thanks for your prompt answer. I would preffer to send you a private message if possible since the domain is NSFW, but i’m having a hard time finding how to do so.

Would you care to explain how i could pm you? On the other hand if there is an email i could send this information then better yet.

Alternatively the #2223335 support ticket has all the information you require (This one has everything you need should you require to forward it to the Bot Management Team, including domain, IPs or rays): Cloudflare Help Center

Best regards,
Luis

#5

Perfect!

We will respond to the ticket!

#6

Hi Luis,

Thank you for the ticket number, I’ve replied to you directly.

To provide some background, we have many checks to identify a bot as a known non-malicious bot. One of the most important checks is how the developer tells us to check, in the case of Bing:

How to Verify Bingbot - Bing Documentation

Verifying Bingbot

If you see what appears to be Bingbot traffic in your server logs based on a user agent string, for example Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm), and you want to know if this traffic really is originating from a Bing server, you can take the following steps:

  1. Perform a reverse DNS lookup using the IP address from the logs to verify that it resolves to a name that end with search.msn.com
  2. Do a forward DNS lookup using the name from step 1 to confirm that it resolves back to the same IP address

In this case, the IP that is not blocked, returns the correct Reverse DNS records:

❯ dig -x 40.77.190.145 @1.1.1.1 +noall +question +answer
;145.190.77.40.in-addr.arpa.    IN  PTR
145.190.77.40.in-addr.arpa. 3600 IN PTR msnbot-40-77-190-145.search.msn.com.

❯ dig msnbot-40-77-190-145.search.msn.com @1.1.1.1 +noall +question +answer
;msnbot-40-77-190-145.search.msn.com. IN    A
msnbot-40-77-190-145.search.msn.com. 3600 IN A  40.77.190.145

However, the blocked IP, does not return reverse DNS records:

❯ dig -x 40.77.139.51 @1.1.1.1 +noall +question +answer
;51.139.77.40.in-addr.arpa. IN  PTR

Even if their online tool says that this is a verified bot, it’s not meeting their own requirements.

So, the best next step would be for you to reach out to Microsoft/Bing’s Support team to check why the blocked IP is not correctly advertising itself through reverse DNS records, as it’s likely a bug/defect in their crawlers.

I hope this helps, let me know if there are any questions.

1 Like
#7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.