I set up a firewall rule that states if not the United State and not a known bot then Challenge. But, on Facebook, the preview started showing “Attention Required! | Cloudflare”.
To fix this, I added… and not “facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)”. I tried the debugger, and this seems to work. However, is this the best way to fix the problem?
I run a very local website that does not seek or need traffic outside of the U.S. and it is starting to get hammered by Countries labeled T1 and Unknown.
T1 is the Tor network, and you might be blocking legitimate local traffic. Might be. Probably not if it’s a mom and pop or local organization type site where your visitors most likely aren’t using Tor browsers.
I also expected Facebook to be a known bot and really didn’t expect their crawler to be outside the U.S. but then I was getting “ Attention Required! | Cloudflare ” in the Facebook preview when posting.
After being up a year, the large traffic from T1 and Unknow locations is very new and spontaneous.
Also, there has been a pattern of an IP (or group of IPs) starting to hammer my site, I block it/them, Firewall Events will log the rapid-fire hits blocked for weeks after then it suddenly stops and a new IP (or group of IPs) will start hammering the website… almost exclusively from outside the U.S…
Are there any ideas for a better Firewall Rule? It is a hyperlocal website. So, I don’t really have a problem having those outside the U.S. being challenged. But, I don’t want to block legitimate bots.
My Security Level is High and has always been set to High. The rapid-fire hits were and do come at that level.
Also, the block of all non-US IPs just started today. Beforehand, I had a Firewall Rule that would completely block specific countries if they were a particular problem or I would create an Access Rule for a specific IP or IP Range that was consistently an issue.
I created the non-US rule today because the T1 and XX traffic is very new. The logged Firewall Events are showing these T1 and XX hits coming through Cloudflare Data Centers mostly in the U.S. but I don’t know if that actually proves anything.
We have one site with private regional traffic and no need for other browsers. Adding User Agent Blocking for the following should catch most of your unwanted Tor traffic until a new version is released.
“I shouldnt have said that, I shouldnt have said that”
The way I understand your rule you want to challenge everything that is not a US bot. Is that really right?
I’d first try to clarify with Cloudflare why cf.client.bot does not seem to return true with Facebook. If it did, your expression should evaluate to false and not challenge.
Your user agent approach generally works, but opens the possibility to circumvent the challenge by spoofing the user agent. As a temporary workaround (until the cf.client.bot thing is clarifed) I’d probably allowlist the ASN 32934 instead for the time being.
