Bots go to the site, the rule is ignored

Made a rule so that the bots wouldn’t pass but try to pass the captcha.These bots get to our site.
AhrefsBot/6.1
MSN
BingPreview/1.0b
semrush.com

What to do? How can I not miss themI do not want to block yet. It will be extreme measures

1 Like

Who will answer ??Support You where I have a crash of nasty traffic with these values ​​in user-agent
AhrefsBot/6.1
MSN
BingPreview/1.0b

1 Like

That’s a pretty restrictive set of rules and it should block most everything. It could be that the bots are bypassing Cloudflare, but it’s not likely if those are legitimate bots. What’s the domain?

1 Like

domain pechatberry.ru

1 Like

The “not Russia” part sure works. So I can’t test any of the User Agent restrictions.

Have you firewalled off all traffic at your server that doesn’t come through Cloudflare?

1 Like

A lot of requests come from Bing yahoo and Microsoft
Tired of these bots as flies flew into my site

1 Like

Right now allowed bots trigger rules in your FW. Firewall rules are evaluated in priority order… Do you have a RayID associated with the requests you say are getting through? What rule is associated with those requests?

2 Likes

Really? There’s an Allowed Bots rule before the one posted? That would certainly make sense.

1 Like

From rule ID A8a3ad49dea64307b8c702d97d0146b4 falls under rule Rule name Enable - Bots

There is a FW rule to enable all bots, and there is also a rule to prohibit others where there is a part of the user-agent string

1 Like

Rules are evaluated in priority order. The sallow bot rule is above (higher priority) than your captcha rule. Based on your described desire you need to reorder them so the rule pictured is a higher priority.

Should it work like that?

Just throwing it out that you should also update your robots file to block each of those bots specifically. Or if you want them to access slowly, add a crawl delay. All of those listed will listen to the robots directive unless it’s someone spoofing one of the bots.https://www.robotstxt.org/

This topic was automatically closed after 30 days. New replies are no longer allowed.