I’d like to ban Grapeshot Crawler for not complying with my robots.txt rules. The user agent is shown as ‘Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)’. In the user agent firewall rule, do I have to use that whole string, or can I shorten it to ‘grapeshot’ or ‘asterick grapeshotasterick’ where asterick = * ?
You can certainly create a Firewall Rule that blocks User Agents that contain grapeshot and that should do it.
Thanks sdayman. Trouble is, I’ve already used up my five free firewall rules. I’d like to put this one under the section for User Agent Blocking, where I have not used any of the 10 free rules yet. User Agent Blocking doesn’t have the ‘contains’ option.
Copy/paste the entire string for a UA blocking rule. That should do it. Watch your logs for any ‘changes’ in their UA string and block any that may appear using those possible stings.
Thank you both!
This topic was automatically closed after 30 days. New replies are no longer allowed.