SEMRush and Ahrefs bots whitelisted but still not able to access site

My team and I have tried everything to enable access for SEMrush and Ahrefs site audit bot but both fail and we get back 301 and 403 errors on the robots.txt file.

  • we’ve whitlisted all 348 SEMrush IP addresses in security WAF tools
  • removed all wordpress plugins
  • uploaded new .htaccess and robot.txt files

Are you hitting the Cloudflare WAF, if so what rule?

You can search by rayID in your security events to see if it is Cloudflare.

https://developers.cloudflare.com/fundamentals/reference/cloudflare-ray-id/#security-events

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.