Cloudflare blocking Ahrefs bots/IPs from crawling

As the title says also I add all this list

and its still not working… I saw few post on that but none of them with solutions and they have been few months ago

Have you checked the Firewall Events Activity Log?

1 Like

yes, all what block come from another countries that not related to ahrefs I also talk with them about it. it was in one account, in another account the firewall events are empty…

I was also try to approve this countries and run crowl again but still was the same problem so i cancel my changes.

I know Cloudflare doesn’t block Ahrefs by default. In fact, Ahrefs is on Cloudflare’s “Good Bots” list:
https://developers.cloudflare.com/firewall/known-issues-and-faq#bots-currently-detected

You need to turn off the rules you’ve added, or settings you’ve changed, that are blocking Ahrefs.

2 Likes

The only things that I did is to add the list of a hrefs IP to be allowed nothings beside that was change…

Then I don’t understand what this means:

Cloudflare really doesn’t block low-level traffic by default. If these blocks are in the Firewall Events Log, what setting was causing it?

If there’s nothing in the Firewall Events Log, then the block isn’t coming from Cloudflare.

add the list of a hrefs IP to be allowed- Base on the ahref link in my first message here, in cloudflacre there is IP Access Rules I add them to be Allow as ahref request.

this is the firewall to these events.

and this is the error from ahref of crawls " The HTTP server returned error 406: “Not Acceptable”. This request was likely filtered by a firewall."

May I ask and suggest, due to the string part Ahrefs in the User-agent, you could try with a Firewall Rule like if user-agnet contains ahrefs and the action allow. That way, eitehr if they change the IP addresses, it should still pass.

  • hopefully, someone does not abuse their user-agent, which could possibly allow some “Bad crawler” to have a part of “ahrefs” in it’s user-agent name while it’s not the real AhrefsBot to expect …


I did this and try to run but it was not working… I also tried it with ahrefs without capital A

Maybe my suggestion is unlikely the cause of issue, but did you proxy :orange: the subdomain?

1 Like

:thinking: Which subdomain?

The subdomain that Ahrefs is trying to crawl.

I must have missed that part of the conversation. I figured it was the main site. But in either case, that would still imply it’s being blocked by the origin.

3 Likes

you did not miss anything is the mean domain, not any subdomain, and it’s worked until last week, this week I saw that was an issue but we didn’t touch anything in Cloudflare for more than a few months…
and there is no any proxsy

I have another account on Cloudflare and on ahref that its working i check tab by tab and all look similar…

it solved the issue was on my server company they did some update maybe it was the firewall or something just to update you thanks for the help

1 Like

I am having all kinds of issues with adding Ahref IPs and their bots. I am not getting a deep crawl.

Is there a bulk tool to add all IPs as they have a long list or do you have to do it for them manually?

Also, I am running a photography site would upgrading to the edge network, WordPress plugin do much good for me? It’s showing me only a few milliseconds of improvement on my desktop performance.

Thanks in Advance!