How to block all good bots except google

I would like to block known good bots except google (and maybe baidu)

I know the firewall has a rule to block good bot but it’s all or nothing, I can’t choose which bots to block and which to let through.

Is there a way to to block known good bots and let google in?

Thank you

A start could be

(cf.client.bot and not ip.geoip.asnum in {15169 55967})

This would block known crawlers except for when they come from Google’s and Baidu’s networks. I believe 15169 and 55967 are their main ASNs, but you might want to check that and include any possible other applicable ASNs too.

You might also be interested in this

2 Likes

I just checked the firewall logs and it blocked baidu spider
so a better question would be how to block specific bots ? Is there a ASN list available somewhere?

Hence exactly what I wrote about adding other ASNs.

You can block them on their individual parameters. That can be by IP address, user agent, ASN, and so on.

This topic was automatically closed after 30 days. New replies are no longer allowed.