How can I configure Cloudflare to allow crawler request from certain IP's like Ahrefs?

I am facing a problem in Cloudflare where crawling requests from external bots at Ahrefs/Semrush are being blocked by Cloudflare as part of std. blocking procedure. For this Ahrefs have provided me an extensive list of IP Addresses which needs be whitelisted for Ahrefs to pass through Cloudflare easily.

Considering the extensive list, I was planning to create an IP List and allocate it at once in rules, but I am unable to do so as I see no such option to allocate IP list and instead every IP Needs to be entered and whitelisted manually in Cloudflare.

Is there convenient way to achieve this in shor and quick way?


First, you will need to determine why this bot is being blocked.

Do you have Bot Fight Mode or Super Bot Fight Mode on?

Ahrefs is already a “Known bot” ( Cloudflare Radar) and shouldn’t be blocked by default so we need to find out why this is happening.

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.