Bot/crawler for websites protected by Cloudflare

Hello,

I need to verify informations provided by users of our app by requesting provided URL and checking if it contains specific data.
Requested websites are NOT our, so there is no possibility to add IP in CF firewall configuration.
During this process, CF blocks some of our requests.

Is there any possibility for something like certification of external APP for GLOBAL whitelisting our IP, like in case of Googlebot/Bingbot and others?

Up up up.

I’m tagging @MoreHelp, as it’s been over 72 hours and no responses.

But to summarize, Cloudflare doesn’t block regular bots. More likely, your clients have chosen elevated security levels that are breaking your service. Your best bet is to inform your clients that they need to add an exclusion for your bots if they want to use your service.

There’s a way to ask Cloudflare to put your bot on their good bots list, but I can’t imagine your odds are good for making that happen. Right now, I can’t track down the link for asking.

Here it is: Cloudflare Bot Verification

5 Likes

@sdayman 's suggestion would be correct here. I would suggest submitting that form with the requested details so that we can review this further.

1 Like

Thank You.