Blocking Semrush crawler

Website, Application, Performance
1

I have been using Semrush for the last 4 months and all of sudden Cloudflare is blocking the Semrush crawler for the last two weeks. This is happening on both my websites in Cloudflare.

I’m getting a 403 error for both websites when the websites are crawled.

3

Can you see / identify the blocks in your WAF events logs?

1 Like
4

How do I do that?

5
6

I think I fix it.

how do I remove the following error. " No redirect or canonical to HTTPS homepage from HTTP version" It is recommended to do the following " Do either of the following:

  • Redirect your HTTP page to the HTTPS version via a 301 redirect
  • Mark up your HTTPS version as the preferred one by adding a rel=“canonical” to your HTTP pages

I don’t know how to do this.

1 Like
7

You could see if the Always Use HTTPS method works for you. There is also information on how to use a Redirect Rule instead.

8

I used the alway use HTTPS method .

I also had a Forwarding URL (Status Code: 301 - Permanent Redirect, Url: for www mysite dot com

I change SSL/TLS encryption mode from flexible to full.

I still have the error

9

If you can share the domain name, we can take a look and see if we can see any adjustments that may help.

10
11

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.