Allowing AHREFS Crawler

I’m trying to allow for the site ahrefs to properly crawl my site to track statistics. I’ve gone ahead and added their IP range correctly (I think). Under firewall → firewall rules I added all 63 of the IP addresses as “IP Source Address” “equal” “IP Address” “or”. I did that 63 times, but when their bot checks the status of my robots.txt file it returns this error still -

The HTTP server returned error 406: “Not Acceptable”. This request was likely filtered by a firewall.
This website is protected by the Cloudflare firewall

Am I entering the IP addresses in the correct place in Cloudflare?
Am I supposed to enter it under Firewall → Tools? If so how would I enter the range? I tried entering (maybe it’s formatted wrong) and it said it wasn’t accepted

Also, looking at the article below would I need to add the ~60 individual IPs starting w/ 168 as well? Any help would be appreciated as I’m new to all this.

Cloudflare allows bots such as this by default. It’s possible some other setting is blocking this, such as Super Bot Fight Mode.

Check your Firewall Events Activity Log for an entry that corresponds with Ahrefs then click on it. It should show you the setting that blocked the crawl.

Super Bot Fight mode is off and when I look at the activity log I can see 4 instances of the ahrefs ip address hitting and the action taken being allowed. When I check on ahrefs however I see the following. Could it be an issue with my host?

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.