Super Bot Fight Mode and sitemap.xml

Security
#1

Hi

I am on the 20$ plan and when I try to access https://www.edityouraudio.com/sitemap.xml I get a captcha page

When I try to get this page, I also get a 403

Is there a way to prevent that ?
Will Google manage to scrape this sitemap anyway ?

Thanks

#2

i just removed the " Definitely automated" “challenge” to “allow” and I can access it again

How can I be considered “definitely automated”

#3

It’s quite possible that the simple act of browsing that specific file makes you seem like a bot because (most) humans would never look at that file.

#4

Thanks but still I had a 403 on the Google Search Console, which seems really bad

#5

That’s definitely a bot.

1 Like
#6

MM the bot that is supposed to make your website discoverable to the world

Google search console should thus be a white listed bot in cloudflare.

Is it possible to mark it as such ?

#7

Super Bot Fight mode isn’t very forgiving. Many people who try to use end up turning it back off because there isn’t (yet) a way to add overrides in Firewall Rules.

1 Like
#8

Weel it seems that it is not useable then

1 Like
#9

It isn’t, but no one at Cloudflare cares. Upgrade to Enterprise for $4,000+ a month then they care.

1 Like
#10

Thanks for the info

#11

I have a basic paid plan and have enabled SBFM and currently have no known problems. I can bring up my sitemap in Chrome browser without any issues or captchas.

I hope these teething troubles can be ironed out

3 Likes
#12

Search crawlers like Googlebot are considered “Verified Bot”, and should not be blocked by Super Bot Fight Mode. Are you sure you don’t have other Firewall Rules that may be blocking or rate-limiting Google? If not then you may want to open a support ticket so Cloudflare support can investigate what’s blocking Google.

The Bot Report includes three traffic types:

  1. Likely automated traffic may have come from bad bots. We use heuristics, machine learning, and other techniques to spot these requests. In most cases, this traffic will hurt your site without providing anything useful in return.
  2. Likely human traffic is legitimate and important. Ideally, the vast majority of your traffic matches this type.
  3. Verified bot traffic comes from good bots on the Internet. We have verified search crawlers like Google and payment notification services like PayPal. Most users choose to allow this traffic.
2 Likes
#14

I have a basic paid plan and have enabled SBFM and currently have no known problems.

Have you checked the Firewall Events log to make sure?

#15

Just did but i only have 72 hours left so can’t see

Maybe I’ll try again later then

Thanks for your help

#17

This topic was automatically closed after 29 days. New replies are no longer allowed.

#18

@martinratinaud

BFM is a sensitive feature and it’s not right for everyone. We offer more configurability with Bot Management Enterprise and can only help BFM customers with basic settings like turning on/off Verified Bots.

If you encounter any issues with this feature (e.g. false positive), we recommend that you disable it under Firewall > Bots:

  • For Free plans, toggle Bot Fight Mode option to Off.
  • For Pro plans, click Configure Super Bot Fight Mode and set Definitely automated and Verified bots to Allow, and toggle JavaScript Detections to Off.
  • For Business and Enterprise (no BM add-on) plans, click Configure Super Bot Fight Mode and set Definitely automated, Likely automated and Verified bots to Allow, and toggle Static resource protection and JavaScript Detections to Off.

For more details, please see https://developers.cloudflare.com/bots/get-started.

3 Likes
Bot Fight Mode for Definite Bots - is blocking Google?