We have recently been attacked by some malicious scripts that are trying to flood our site with queries. We have identified it coming from some less common user agents and so we created some firewall rules to block these user agents, which seems to have generally mitigated the problem.
And in order to prevent the possibility of any false positives on those user agents, we also added something into the blocking rules to cover just the typical queries being sent by the scripts.
It seems though we might have accidentally blocked some of Google’s search bots too. I say this because I received an email from Google Search Console saying that some of the pages on our site can’t be accessed because of a “Server error (5xx)”.
So I went in and turned off “Bot Fight Mode” under the Bots tab within Firewall settings.
I also added another condition to each of the user agent blocking rules for “Known Bots” and then switched the Known Bots condition to “off”. But I am a bit confused whether I added the Known Bots condition in correctly or not in order to exempt known bots from the blocking rules.
Here is an example of what I have for my user agent blocking rules:
User Agent > Contains > “The Bad User Agent”
URL Full > Contains > “The Query Typically Being Sent By The Script”
Known Bots > Off
Thus, I have the following questions please:
1 – Should I disable or enable Bot Fight Mode?
2 – In my example rule above did I add in the “Known Bots” command correctly in order to exempt all known bots from the user agent blocking rules I created?
3 – I also have some Captcha and JS Challenge rules set separately for certain countries. Besides the USA, are there any other countries that Googles sends their bots from which could be affected by these country rules and also be a possible cause of the “Server error (5xx)”?
4 – I noticed under Firewall > Tools that you can also set up some User Agent blocking rules there too, but I have avoided using this function because it seems you can’t add any conditions to the rules, nor can you set up a rule that contains only a portion of the user agent. So I think it is best that I continue to set up these user agent blocking rules within the normal Firewall Rules settings as I have been doing since it offers more flexibility with Firewall Rules in general?