Problems with Ahrefs Crawling

Hello everyone, I’m still not sure if the problem is in Cloudflare, but I want to try my luck here.

So the issue is that websites don’t crawl on Ahref. And the only websites from all connected that don’t crawl are the one connected to Cloudflare, so I assume the problem is here.

Some of websites have partial integration and some full and the same problem.

We created rules that whitelist User Agent - contain - Ahref and all IP addresses that were provided.

The Issue we get when trying to crawl it is:

No seeds found among the URL sources

I will provide screenshots for one of websites. With settings and error.

Also, I tried to disable Modsecurity, talked with Ahref Support, checked forums and no solutions yet. Can someone help me solve this issue?

Screenshots:

It’d sure be nice if they could map out those redirects for you. Firewall won’t redirect. It’ll just hit the brakes.

So what can help with these redirections? Because other websites, that are not connected to Cloudflare, don’t have this issues

Other issues aside, you could likely simplify such firewall rules with something like this:

(http.user_agent contains "ahrefs" and cf.client.bot)

Ahrefs is on Cloudflare’s known-bots list.

1 Like

Thanks, I still get the error with redirections. And can’t check if it works or not.

Here is the screenshot of what I get when trying to audit website, maybe this will help:

Requesting for more help @More help
@MoreHelp

I still didn’t get any answer, so maybe Cloudflare Team can help? @MoreHelp

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.