Whitelist Crawler AHREFS not working AT ALL

Hi,

I am going nuts trying to whitelist the Ahrefs crawler on Cloudflare.

I followed these steps

  1. Add the Ip range of Ahrefs (link here) into the firewall section.
    –> Doesn’t work // Here is the error message on ahrefs :
    "The HTTP server encountered an internal error (520). The server may currently have issues.
    This website is protected by the Cloudflare firewall"
  2. Tried to add a firewall rule, with the same ip adresses.
    –> Still doesn’t work
  3. Tried to “Pause” cloudflare on the overview page
    –> Still same error

Please can someone help me I paid for ahrefs crawl and can’t do it, that’s really annoying.
Thank you for your help :slight_smile:

That would suggest you didnt wait long enough for DNS propagation to take place, as a 520 should only come from Cloudflare.

A 520 should not indicate a block however, but rather a connection issue with your server - > https://support.cloudflare.com/hc/en-us/articles/115003011431-Troubleshooting-Cloudflare-5XX-errors#520error

Whats the domain?

Hi Sandro,

Thank you for your (very) quick reply !
My domain is french, (o2 Switch

I looked on internet and I understood that they already have cloudflare ip whitelisted.
Then I asked them to be sure to whitelist also the Ahrefs ip adresses…

So if I understand well, I have 2 problems, one is the cloudflare firewall still active, even if I whitelist IP adresses or put my full website on pause (I get that I need to wait a bit longer :slight_smile: How long appprox ?), and the other one is my origin server, which will get a problem anyway… Right ?

1 Like

At this point I am not even sure it is a firewall issue at all, as you only seem to get 520s, which point at a connection issue rather than firewall one.

Once that is fixed, there certainly is the chance the firewall might intervene too, but I believe it is too early to say that yet.

However your domain does not point to Cloudflare and appears to never have properly validated for Cloudflare in the first place. You’d first need to correctly set up your domain on Cloudflare.

Which nameservers were you given?

Oh…:confused:
The nameservers I was given were : darl.ns.cloudflare.com // mimi.ns.cloudflare.com
I don’t understand why you say that, I never have any alert or something telling me that my config is not working. Can you tell me what’s wrong ?
By the way, thank you :wink:

Currently your domain does not point to Cloudflare, hence it will not work in the first place. You first need to configure your domain properly. You still have two o2sdns.fr nameservers configured.

Can you post a screenshot of your Overview screen on Cloudflare as well as one of the “Cloudflare nameservers” section on your DNS screen?

It would seem even the assigned nameservers do not know about the domain.

Ehm, your domain is o2switch.fr, correct?

Yes, it’s correct.
Here is a screenshot of my overview cloudflare :


Ehm, no. Your domain is dermelia.fr.

Cloudflare actually recognises the crawler in question and I wouldnt expect them to block it unless explicitly instructed to

And again, that error does not suggest a block but rather a connection issue. At this point my assumption would be there might be temporary issues with your server, but that is something your host would need to clarify I am afraid.

Ok thank you for your clarification, So do you confirm I am ok with my cloudflare configuration ?
I will wait for my server provider answer

Your Cloudflare configuration looks all right. Just one thing, make sure your SSL mode under “SSL/TSL” is set to “Full strict”.

1 Like

Also make sure your host has the following addresses whitelisted, just in case he applies some rate limiting or address blocks