I see several thousand requests by some bots pretending to be user-browser agent on my domain. I find Cloudflare is unable to tag them under known-bots category because of user agent string pretend to be real, even though in some cases I see version of browser is way older than the latest.
Is that possible to setup some sort of rule in cloudflare to reject/challenge the traffic with browser-user-agents having outdated browser version/build mentioned under it ? If yes, can you please suggest how I can go for it ?
I just thought of having WAF rule with user-agent filter (RE based) to challenge any such traffic but I could never make RE works in Cloudflare myself for comparison of browser build range.
All modern browsers now started doing auto-update on regular basis and so I am not sure if any real users can pass requests with older browser user agent strings.
Cloudflare bot management module - is it capable of detecting such smart-bots and challenge it ?
May I ask, if you see them in Firewall Events, have you created a User-agent blocking rule, or even better, a Firewall Rule which contains a “string” for that specific user-agent to block all the requests comming from them?
May I ask you to share the user-agent or any other non-private information here, so we could help you to create your Firewall Rule to block them?
I think it could depend on the selected settings and a Cloudflare Plan which you use, just like in combination with Privacy Pass, Browser Integrity Check , default Security Level (at least Medium), Bot Fight Mode option, etc.
You can even block requests to some of the compatible Cloudflare ports, even specific countries, IP address(es), whole ASNs, etc.
No - actually I had noticed them after had a firewall rule tracking all request on particular host under the domain. Yes that`s one way of doing but I did not want to track all requests from old browser agent string by manually defining what are all old user agent strings. Rather I look forward to define minimum acceptable version of browser and have users follow it possibly…
Here are some examples of user agent strings from old version of browsers:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/66.0.3359.139 Chrome/66.0.3359.139 Safari/537.36
Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36,gzip(gfe)
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 YaBrowser/184.108.40.2060 Yowser/2.5 Safari/537.36
I do understand that sometimes, users dont set browser update automatic, and so might be actual users trying to hit the website but I dont want to deny the fact that still users can automate the traffic pretending to be a browser from load testing/crawling tools.
We have enterprise licensing currently at our domain but I could not see any option under bot fight mode which can recognize the pattern based traffic and challenge it.
Yes - this is an option I thought of as last thing, we just wanted to keep the site available public in all but if bots are draining performance, maybe blocking thru ASNs OR regions might be a good option.
This is awesome. It can really help making solid rule which at least can stop traffic from all of these defined bots.
This is a very common technique that bots use, It should be a standard on the enterprise range but I have not had the chance to test bot management from an enterprise account yet.
Typically those bots will fail to spoof other components of the browser, such as the JS engine, the window resolution, and a bunch of other client-sided integrity checks that you can’t really configure on the Cloudflare panel. Overall they are very easy to detect and mitigate but you need to inject some code on the client to catch them.
Edit: IIRC the bot management from Cloudflare has TLS fingerprinting, if that’s the case and you are able to upload your own fingerprints to block, it would be more than ideal for this case scenario.