fritex
October 30, 2021, 11:34am
2
Hm, that would be really nice but I think, the normal requests from normal visitors would be JS Challenged as well, which isn’t a good thing, right?
Either one rule could end up being in a conjuction with the second one - I am afraid how that would either work, if so.
Could be I am wrong about it.
I would rather suggest below:
Known “good” bots are listed on the below link, kindly check which one are they:
To block bad bots, I would suggest creating another Firewall Rule which includes a list of them, something as follows on below posts:
Here’s a list from the perishablepress.com 7G .htaccess firewall:
(360Spider|acapbot|acoonbot|ahrefs|alexibot|asterias|attackbot|backdorbot|becomebot|binlar|blackwidow|blekkobot|blexbot|blowfish|bullseye|bunnys|butterfly|careerbot|casper|checkpriv|cheesebot|cherrypick|chinaclaw|choppy|clshttp|cmsworld|copernic|copyrightcheck|cosmos|crescent|cy_cho|datacha|demon|diavol|discobot|dittospyder|dotbot|dotnetdotcom|dumbot|emailcollector|emailsiphon|emailwolf|exabot|extract|eyenetie|feedfinder|flaming|…
Good list, thanks. I have deployed that but removed python and demon (those seem to block some RSS feedreaders, YMMV).
What I also have in place is this:
(http.user_agent contains “SemrushBot”) or (http.user_agent contains “AhrefsBot”) or (http.user_agent contains “DotBot”) or (http.user_agent contains “WhatCMS”) or (http.user_agent contains “Rogerbot”) or (http.user_agent contains “trendictionbot”) or (http.user_agent contains “BLEXBot”) or (http.user_agent contains “linkfluence”) or (http.us…
You could also combine multiple things (like bad bots) into a single Firewall Rule, like block all bad bots but at the same time allow good “known” bots from the cf.client.bot
list.