Cloudflare blocks my good bot

General
#1

Cloudflare blocks my good indexing bot from indexing pages that are supposed to be accessed by indexing bots.

Even more, it blocks my bot even from accessing robots.txt! The site in question is https://www.biorxiv.org/robots.txt

My bot indicates that it’s a bot in its user-agent string with “prophy-bot/1.6”, it respects Crawl-delay, etc.

It says on help pages https://www.cloudflare.com/en-gb/learning/bots/how-to-manage-good-bots/ that Cloudflare recognizes good bots and puts them into allowlist, but apparently it has not learned about mine yet. Can this be fixed?

#2

The “Good Bots” in the allow list are a tightly regulated list.

Generally, Cloudflare doesn’t block bots unless you’ve activated some sort of bot-blocking rule. Can you check the Firewall Events Log in the Cloudflare dashboard to see why your bot is being blocked?

#3

I’m not a Cloudflare user myself, so my dashboard is completely empty.

#4

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.