SEMRUSH Bots whitelisting

Hello, :slightly_smiling_face:

  1. We are using SEMRUSH for our site(s) SEO and Marketing Audits. We added Cloudflare on our site recently and since then the SEMRUSH crawler is forbidden to our website. We came to learn that we need to Whitelist their IP Addresses with Cloudflare Firewall. We did as per their article but despite adding them on the Firewall it’s not working.

  2. We also tried to block a specific country from which we don’t want traffics from, that also not working at all.
    Here’s the screenshot: Screenshot by Lightshot
    Looking forward to a quick solution.
    Thanks upfront!

I’m assuming these were put in the IP Access Rules, correct?

Do you have other rules in the Firewall Rules, Zone Lockdowns, IP Access Rules, etc. that may be causing a block? Can you find examples in your logs that show the action taken?

Hello,
Thanks for your reply.
Yes, we have added the 3 IPs under the ‘Tools’ whitelisting section as per our screenshot.
Other than that we were able to block a specific country from our Firewall Rules Section.:slight_smile:
I also added the SEMRUSH IP Addresses with an OR combination and marked them ‘allow’
Here’s the screenshot: Screenshot by Lightshot
Hope I have done it right?

Check for the order number of that rule (priorityof execution) - maybe you block countries first, then allowing Semrush bot to access your domain/website?, or vice-versa. Which could result also as a return like semrush bot being blocked.

That screensho is from IP Access Rules, right?

Moreover, SemRush bot gets to my sites without blocking. Have you checked your Firewall Events Log for this access atempt of Semrush bot as being blocked somehow?

From my point of view, actually not so good but could work.
Meaning, you allowed Semrush bot IP addresses, which could result as if they get compromized, all the requests are trusted and would go straight to your website/domain.

Better approach would be to select “bypass” option (to bypass the security options) for that Semrush bot, if you have some security settings setup and bot fight management enabled for example, and/or some other Firewall rules to block other “seo” crawlers (just in case).

Have you tried setting up and allowing Semrushbot to crawl your sitemap file at your robots.txt file too?

An article for that would be:

I tried all the settings, added the SEMRUSH bots to whitelist, bypassed them but no result. I also updated on robots.txt. I can’t attach the log due to guidelines.

Yes, that’s the setting.
I tried all the settings, added the SEMRUSH bots to whitelist, bypassed them but no result. I also updated on robots.txt. Here’ the screenshot of the error log.

Looks like they don’t let me post the link. :frowning: