Need a basic setup where the major “known bots” are allowed to crawl my site to index new content and at the same time to be able to challenge France(with captcha ) when sending a GET request…
Also as far as “known bots” do I really have to open the firewall for “ALL” of them or just the big ones… And which are considered the big ones? Thanks.
The bots list below is part of the dev docs for Firewall Rules. That should point you in the right direction for creating your firewall rule. Go ahead and post a picture of your completed rule if you have any questions.
looks like I replied to myself… Please see picture in conversation. Thanks.
I do not think that User Agent check will work. I’d go with a series of Does Not Contain ___ AND Does Not Contain ___