Configure Trusted Bots

I am worried my site is being scraped, and I would like to configure the Definitely automated parameter of the Configure Super Bot Fight Mode to at least Challenge (if not Block). The problem is that I have my own internal programs and monitors that call my own API’s that are for all practical purposes bots. Is there a way I can allow “trusted” bot traffic (traffic from IP addresses that I know my programs are running on)? Is there a way to configure a Firewall Rule that allows trusted IP’s and won’t be challenged by the Bot Fight Mode setting?

Unfortunately, there’s not yet a way to create exceptions to Super Bot Fight Mode.

You can create Firewall Rules that return a JS Challenge for threat scores, ASNs, Countries, etc., that are giving your trouble.

But if my own software hitting my own api looks like a bot, is there any way I can have my software answer the challenge that is being generated? Or is there any way for me to pass an encrypted key or something that cloudflare will recognize as legitimate and allowed.

It seems unfortunate that my options are (1) allow all untrusted bots and have my own valid sofware or (2) challenge or block bots but be unable to build any software that uses my own apis.

You could ensure your server only accepts connections from Cloudflare IP addresses ( IP Ranges | Cloudflare) and your service’s IP address (via a firewall on your provider or your server). Once this is done, configure your service hosts file to point to your domain directly, bypassing Cloudflare (assuming your service is trusted).

You can then turn on SBFM as indicated as all valid traffic will come through Cloudflare only or from your service.

Note this means your service will not benefit of any Cloudflare feature in this configuration.

Mind you this will affect other automated services but yours will have direct access.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.