Semrush bot not able to crawl

Hi everyone, I’m having trouble giving permission to the semrush bot to crawl our site.
I’ve whitelisted it with the IPs that semrush support gave me but still it can’t access.
Anyone had the same problem and can help out please?
Thanks!

Nacho, I’m having the same problem… did you find a fix?

Have created any firewall rules, are other bots able to crawl your site and do you have an entry in your robots.txt file that would prevent it?

HI Martyn, you have to contact the semrush support team and ask them to give you the IPs in order to whitelist semrush’s bots. Then on Cloudflare you have to set up the rule on firewall / tools and you put there the ip and which website will affect
Hope that hepls
N

1 Like

Did you check your robots.txt file?

Hi Withheld,

I have created firewall rules to allow the ip addresses SEMrush provided.
I have tried to allow the User-Agent the provided.
We don’t have a robot.txt

It’s still getting blocked.

Cheers,
Martyn

Hi user3011, we do not have a robot.txt file.

N, thanks for your reply, but I’m still struggling on my end.

I have created firewall rules to allow the ip addresses SEMrush provided.
I have tried to allow the User-Agent the provided.
We don’t have a robot.txt

It’s still getting blocked.

Cheers,
Martyn

Ok, I would contact the semrush support team then
Good luck!

This topic was automatically closed after 30 days. New replies are no longer allowed.