Firewall Blocking SEMrush

I added cloudflare a few weeks ago, and the minute I switched 5 of my sites to run through Cloudflare, SEMrush started reporting that they were being blocked from crawling those 5 sites. I went into the firewall settings and whitelisted the SEMrush IP addresses, but it is still showing that they are being blocked. This is the error log I was provided by SEMrush:

IP addresses of Semrush Bot ( Site Audit ):,,

curl -i -sS -L --proto-redir -all,http,https --max-time 5 -A ‘Mozilla/5.0 (compatible; Googlebot/2.1; +’

HTTP/1.1 301 Moved Permanently
Date: Tue, 23 Feb 2021 23:01:25 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Wed, 24 Feb 2021 00:01:25 GMT
cf-request-id: 0872b9510c0000c1841b039000000001
Report-To: {“max_age”:604800,“group”:“cf-nel”,“endpoints”:[{“url”:“”}]}
NEL: {“report_to”:“cf-nel”,“max_age”:604800}
Server: cloudflare
CF-RAY: 62649194dfd6c184-IAD
HTTP/2 301
date: Tue, 23 Feb 2021 23:01:30 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=d8dbc07804f0851b2853d65b24926f44d1614121285; expires=Thu, 25-Mar-21 23:01:25 GMT; path=/;; HttpOnly; SameSite=Lax; Secure
x-redirect-by: WordPress
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
cf-cache-status: DYNAMIC
cf-request-id: 0872b9514b0000f198a4863000000001
expect-ct: max-age=604800, report-uri=“
report-to: {“group”:“cf-nel”,“endpoints”:[{“url”:“"}],"max_age”:604800}
nel: {“max_age”:604800,“report_to”:“cf-nel”}
server: cloudflare
cf-ray: 626491954eaff198-IAD

Any ideas to help with this?

That is not a block. That’s a run of the mill redirect. They should know better.

They’re not crawling the canonical URL - they’re starting with http to the apex domain, so the first redirect is to https, and the second redirect is to www.

So how do I fix this? Do I just need to call SEMrush and tell them to fix it, or is there something I can do to make this work?

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.