Firewall Blocking SEMrush

I added cloudflare a few weeks ago, and the minute I switched 5 of my sites to run through Cloudflare, SEMrush started reporting that they were being blocked from crawling those 5 sites. I went into the firewall settings and whitelisted the SEMrush IP addresses, but it is still showing that they are being blocked. This is the error log I was provided by SEMrush:

IP addresses of Semrush Bot ( Site Audit ): 46.229.173.66, 46.229.173.67, 46.229.173.68

REQUEST:
curl -i -sS -L --proto-redir -all,http,https --max-time 5 -A ‘Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)’ http://biznification.com/

RESPONSE:
HTTP/1.1 301 Moved Permanently
Date: Tue, 23 Feb 2021 23:01:25 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Wed, 24 Feb 2021 00:01:25 GMT
Location: https://biznification.com/
cf-request-id: 0872b9510c0000c1841b039000000001
Report-To: {“max_age”:604800,“group”:“cf-nel”,“endpoints”:[{“url”:“https://a.nel.cloudflare.com/report?s=BednRXHuAb0FhZmVPzLEpKhJZ6aK0m9MUJyv9s8P7yo%2BAZBaFExlNUEo5i6CdWPyD8ciV4HDAaHSwiTxQcczbLtaIvodmXr2HOUZO5xBb32JEg%3D%3D”}]}
NEL: {“report_to”:“cf-nel”,“max_age”:604800}
Server: cloudflare
CF-RAY: 62649194dfd6c184-IAD
HTTP/2 301
date: Tue, 23 Feb 2021 23:01:30 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=d8dbc07804f0851b2853d65b24926f44d1614121285; expires=Thu, 25-Mar-21 23:01:25 GMT; path=/; domain=.biznification.com; HttpOnly; SameSite=Lax; Secure
x-redirect-by: WordPress
location: https://www.biznification.com/
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
cf-cache-status: DYNAMIC
cf-request-id: 0872b9514b0000f198a4863000000001
expect-ct: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct
report-to: {“group”:“cf-nel”,“endpoints”:[{“url”:“https://a.nel.cloudflare.com/report?s=DZIgNoBhKQbIdyWRkx09OKfMa1gW59pufy5L7J3X%2FpUgsHsAJEZ6Arsw0%2BIpZY7fieqRuhpHzM%2BzKeFqxdBmURfebAx4XXbKNaM6zIfB6tDB7A%3D%3D"}],"max_age”:604800}
nel: {“max_age”:604800,“report_to”:“cf-nel”}
server: cloudflare
cf-ray: 626491954eaff198-IAD


Any ideas to help with this?

That is not a block. That’s a run of the mill redirect. They should know better.

They’re not crawling the canonical URL - they’re starting with http to the apex domain, so the first redirect is to https, and the second redirect is to www.

So how do I fix this? Do I just need to call SEMrush and tell them to fix it, or is there something I can do to make this work?

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.