What is the name of the domain?
What is the issue you’re encountering
Ahrefs Bots Blocked with 403 Despite allowlisting IPs & User Agents
What steps have you taken to resolve the issue?
Since February 14, we have been experiencing issues with Ahrefs bots crawling two of our websites. Despite following all recommended steps, including allowlisting Ahrefs’ official IP addresses and user agents, the bots continue to receive a 403 status code.
We have added the IP ranges both through Custom Rules and IP Access Rules under Tools, as well as configured additional allowlists via Custom Rules. However, there has been no improvement.
Interestingly, when we disable proxying for the domain, the issue disappears, and Ahrefs is able to crawl the sites without any restrictions.
We would appreciate any guidance on resolving this issue while keeping Cloudflare’s proxy enabled.
Was the site working with SSL prior to adding it to Cloudflare?
Yes
What is the current SSL/TLS setting?
Flexible