Facebook's crawlers - How to rate limit the user agents?

Website, Application, Performance
4

I’ve tracked down quite a lot of requests to my domain with log, therefrom come to the some conclusion for some specific.

Despite it might not work for everyone’s case, however I’d like to share my case and example.

Name of the Rate Limiting Rule I have: IP Path HR

TL-DR;
Since Website mostly for Croatia :croatia: people (10-20k daily visitors), despite it’s a small country, therefore I don’t care for who tries to smuggle some strange requests from the inside country HR since if someone does something, it get’s reported immediately to the provider and CERT team to resolve. Therefrom, anyone outside HR gets Rate Limited for strange behaviour - reasonably, including “crawlers” as well).

In simple words:

  • If X hits the same path Y from same IP Z, 10 times in 10 seconds, gets Rate Limited for 10mins. I am using the “old version” of Rate Limiting Rules

Expression (anyone from outside of Croatia):

  • (ip.geoip.country ne "HR")
  • ip.geoip.country field requires a higher plan, see here what’s possible and available to you (Rate limiting rules | Cloudflare Web Application Firewall (WAF) docs)
  • Followed by the:
    • With Cache status to “Apply rate limiting to cached assets”
    • With the same characteristics “IP” and “Path”
    • When rate exceeds 10 requests per 10 seconds
    • Then take action “Block”
    • With the following behaviour Block for the selected duration of 10 minutes

To rate Limit by the user-agent would require at least a Business plan (source: above link).

You could give it a try with a “URI path” then contains or matches / (http.request.uri.path contains "/") (to match all the requests), or rather use wildcard * (apex):

slika
slika647×563 14.4 KB

slika
slika1111×1143 30.9 KB

Below is what I have daily rate limited with my rule:

slika
slika1038×826 41.2 KB

From which Facebook is part Rate Limited while the other is Blocked fairly:

slika

Sharing and other stuff working normally on Facebook and abroad, no traffic drop or errors shown.

Bonus tip: Keep away the possible “cache busting” way of clicking on the shared Facebook link to your Website.

Back a while ago, I still have it configured with a Page Rules for my zone (which covers sub-domains as well as main domain and www, if any of the zone URLs are shared on Facebook, therefore clicked and a lot of traffic coming to my Website from Facebook, each click has got a hash part in the URL, therefore the content is not cached at the server itself, rather it’s served from the origin host/server which keeps the CPU load up for nothing, despite I do have page cache configured on the WordPress - as an example):

*.domain.hr/*?fbclid=*
Forwarding URL (Status Code: 301 - Permanent Redirect, Url: https://$1.domain.hr/$2)

There is also a way to do it either via Redirect Rules, or via Transform Rules, otherwise using a Worker (but it’ll cost you, better with Rules to do it).

Hope it helps a bit.