Rate limit misbehaving facebook crawler

Hi there

I am on the free plan. Any suggestions on how to slow down the Farcebook crawler. I don’t want to ban it completely, but it’s a massive resource hog and when it hits my site, it usually spikes the server to an enormous load.

I have reported this to Facebook (along with many other people in their farceboog developer bug reports section) however, they always close the ticket saying it’s resolved when it is not.

FB themselves have added a help article here but it seems to have no effect / ignores the robots txt I am still getting hit hard by this thing.

For the time being, I have set up a “challenge” for their ASN AS32934 But that’s not ideal… So can you suggest any help to reign in this very hungry bot ?

I could try the rate limiting tool - do you think that would work with the ASN ? The problem is, I’d have to specify the root ?


sandro via Cloudflare Community

You can certainly use Cloudflare’s tools for that, but that’s taking a sledgehammer to crack a nut :slight_smile:

Well as I already said, the FB recommendations don’t work, so I’m hoping to find another solution

OK I have put some rat limiting actions in place… see what happens

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.