I’m having some troubles with crawling our website using Semrush bot. I’ve already used the IPs given by Semrush and added the IP Access Rules as well but the error keep showing. Nothing wrong with the robots.txt too I believe.
Can anyone help me with this please?
And by the way, do we need to wait for it’s being active?
What issue are they seeing, do you have an error code?
It said that the site audit was failed and it recommend me to whitelist the SEMrushBot.
Here is the notification:
And I’ve already followed the instructions but the issue is still there :’(
If you check the firewall events log in your Cloudflre dashboard at the time they try to visit, do you see a corresponding entry with a challenge/block action? If so, that should help you work out what is stopping it.
I’m afraid that I didn’t see anything :’( just checked