“Known Bots” should be called “Good bots”.
Simply saying “known” does not tell us if you’re referencing the good, bad, and the ugly, or all bots in general. And in people’s minds, ‘bot = bad’.
“Known Bots” should be called “Good bots”.
Simply saying “known” does not tell us if you’re referencing the good, bad, and the ugly, or all bots in general. And in people’s minds, ‘bot = bad’.
Related, Enhancement for crawler/bot matching
That would be a matter of POV; I personally don’t want Yandex/Sogu/Baidu indexing my site.
My site has no application to the area and there is just waaaaayyy tooo much craap out of the area.
Hi,
what rules would you have to block “Yandex/Sogu/Baidu” and allow Google/Bing/Yahoo/Apple? Thanks
There’s no rule for that. You’d have to ban their IP address ranges manually.
they seems to follow robots.txt.
Thanks for pointing this out, Jules. You’re absolutely right that this field refers to good bots (not bad bots!). It feeds from the same directory as our “Verified Bots” field.
We’re beginning to merge “Known Bots” with “Verified Bots” to make things more clear when you write rules. This might take a bit behind the scenes, but should be ready soon.