How to block FB/Google/Bing crawler hitting


#1

Hi, does anyone knows that how to block the crawling from FB/Google/Bing. Those bots are wasting much of our BW and also affect data statistic.So, we don’t know if CF has any function to identify those bots and then block them.

Appreciated


#2

:wave: @gymjs123,

You can do this using a Firewall Rule: https://developers.cloudflare.com/firewall/cf-firewall-rules/

-OG


#3

but with cloudflare in front, you should be saving bandwidth as well so shouldn’t be much of an issue ? Especially, with cloudflare’s announced bandwidth alliance as well https://blog.cloudflare.com/bandwidth-alliance/

Customer Benefit

The Bandwidth Alliance is providing this benefit to all Cloudflare customers at no additional cost. If you’re hosted with a member of the Bandwidth Alliance, your data transfer fees should decrease when the required technical and accounting systems are activated by the member. Some of those systems are live today. Some are planning to go live over the months ahead.


#4

Bots probably aren’t hitting much cached content, as they’re mostly pulling HTML.

Plus…if you don’t want them, they’re annoying.

I did the Block-the-Bots firewall rule for some of my private sites. But added an exclusion for a monitoring bot user agent so I still get uptime monitoring.


#5

Most stats software filter out bots, but still show you that data separately. It’s usually good to know.
Did you make a special data analyzer on your own?

On a related note…
If you’re worried about server resources more than stats, you can limit any initial session to two seconds. Once the session cookie/variable exists you can then reset the session to 30 minutes.


#6

This topic was automatically closed after 31 days. New replies are no longer allowed.