Facebook Crawler fails(403)

Facebook crawler fails with code 403 when trying to crawl the site. Disabling the Cloudflare DNS solves the problem, however, not an optimal solution.
What else can I do?

Facebook crawler is not blocked by default. My guess is you’ve probably set one or more firewall rules that are causing it to be blocked. You should add a firewall rule (or add this element to an existing firewall rule) to allow Facebook, either by AS Number or User Agent.

So if you have, for instance, a rule that challenges visitors from certain countries, you could add to that rule

… AND ip.geoip.asnum ne 32934 THEN Block/Challenge/JS Challenge

Alternatively, you could whitelist Facebook on Dashboard > Firewall > Tools

The above examples use a Facebook ASNum, which will whitelist not only the crawler but possibly other Facebook services. If you want to be more specific, this page gives you some details: https://developers.facebook.com/docs/sharing/webmasters/crawler/

There’s a Known Bots operator, but Facebook and Twitter are not in that list yet:

Hi Floripe,
Thank you for your quick response.
I don’t recall ever messing with the Cloudflare firewall, but I’ll give the ASN AS32934 a go and see if that works.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.