How to block all good bots except google

I would like to block known good bots except google (and maybe baidu)

I know the firewall has a rule to block good bot but it’s all or nothing, I can’t choose which bots to block and which to let through.

Is there a way to to block known good bots and let google in?

Thank you

A start could be

(cf.client.bot and not ip.geoip.asnum in {15169 55967})

This would block known crawlers except for when they come from Google’s and Baidu’s networks. I believe 15169 and 55967 are their main ASNs, but you might want to check that and include any possible other applicable ASNs too.

You might also be interested in this

2 Likes

I just checked the firewall logs and it blocked baidu spider
so a better question would be how to block specific bots ? Is there a ASN list available somewhere?

Hence exactly what I wrote about adding other ASNs.

You can block them on their individual parameters. That can be by IP address, user agent, ASN, and so on.