Cloudflare blocks my good indexing bot from indexing pages that are supposed to be accessed by indexing bots.
Even more, it blocks my bot even from accessing robots.txt! The site in question is https://www.biorxiv.org/robots.txt
My bot indicates that it’s a bot in its user-agent string with “prophy-bot/1.6”, it respects Crawl-delay, etc.
It says on help pages https://www.cloudflare.com/en-gb/learning/bots/how-to-manage-good-bots/ that Cloudflare recognizes good bots and puts them into allowlist, but apparently it has not learned about mine yet. Can this be fixed?