When I enable Cloudflare, SEMrush thinks that http and https are separate pages.
I noticed that there are multiple 301 GET requests, such as:
46.229.173.67 - - [20/May/2019:11:35:11 -0500] “GET / HTTP/1.0” 301 234 “-” “Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)”
which explains why the SEMrush bot crawls both HTTP and HTTPS versions of your site.
The site is https://www.primocraft.com/
Hi @webwizard,
From my experience, I can tell you that the SEMrush crawler is not a good fit for this kind of reference.
Anyway, if it’s okay to serve the site only through HTTPS, solving this problem is an easy task: just enable the Always use HTTPS option in the Crypto tab of your Dashboard. This will cause all HTTP requests to receive a 301 redirect to their respective HTTPS destinations.
Please let me know if you need any further help.
This topic was automatically closed after 30 days. New replies are no longer allowed.