Ahrefs Crawl Showing Random 403 Errors

I’m getting a few random 403 errors in my Ahrefs Site Audit results lately and I think Cloudflare may be what’s causing them.

The crawl runs fine but there are always a few URLs that come back with a 403 error. It’s different URLs every time but the URLs are always accessible when I click on them.

I’ve tested this by pausing Cloudflare temporarily and running a crawl in Ahrefs - when Cloudflare is paused this doesn’t happen so I’m pretty sure it’s a Cloudflare issue.

I have also tried allowlisting Ahrefs IP addresses in “Security” > “Custom Rules”. I’ve added all of the specified IP addresses and selected “Skip” > “All remaining custom rules”.

I don’t know if this fully whitelists all of the IP addresses though?

Any help would be much appreciated!

Any ideas about this?

Help with this issue would be much appreciated!

You may need to set to skip the managed rules as well.

Likely the 403s are from a challenge, you can find out why the challenge was issued by looking in your security event log here…
https://dash.cloudflare.com/?to=/:account/:zone/security/events

If using Bot Fight mode on a free plan, you may need to disable that since it can’t be skipped by WAF rules. Or use a Pro plan where Super Bot Fight mode can be skipped.

Thanks - I’ve now ticked all of the boxes in this section to skip all rules for the IP addresses for Ahrefs but I’m still getting the same issue.

I’ve also disabled my custom rule and cleared the cache but this hasn’t made any difference - the 403 errors still show in the crawl log.

I can’t see anything in the “Events” section that matches up with the times of the Ahrefs crawls, although there are some events which show up when I click on the WAF rule with the Ahrefs IPs which have skipped all rules but these again don’t match up with the times of the Ahrefs crawls.

Very tempted to just turn Cloudflare off! It’s starting to become more hassle than it’s worth…

Ahrefs recommends using IP access rules instead of the WAF custom rules, here. Try that instead…