Excessive Crawling Activity by Facebook Crawler

I am writing to express my concern about the excessive crawling activity of Facebook’s crawler. This excessive crawling is causing significant performance issues and potential downtime for our website.

Our web server logs indicate that Facebook’s crawler (facebookexternalhit/1.1 -
2a03:2880:22ff:7::face:b00c) is making multiple requests to our wordpress website every second , even during off-peak hours. During peak hours, the crawler’s activity spikes to tens of thousands of requests per minute. This excessive crawling is overwhelming our servers and causing them to slow down or even crash.

We understand that Facebook’s crawler is necessary to index our website and make our content available to its users. However, we believe that the current level of crawling is excessive and unreasonable .

Is there any solution to minimise that ?

I am using nginx in my server and Cloudflare Free basic plan.

Rate limiting should do it:

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.