Proper way to deal with the FB bot (hundreds of requests)

I have a newspaper site (on WP), receives around 25-30K visits a day.
From several months to date I’m dealing with hundreds (sometimes thousands) of requests from the FB bot. Some of them are trying to retrieve old posts (which now are blocked) but some are requesting recent posts. It is the usual facebookexternalhit/1.1, it requests the same post or other posts several times per second, for a short period of time or longer (5 to 30 min), once or twice a day.
Also it is a vicious circle because some of the posts are published in the FB page
The thing is that I can’t just block the bot because the preview (when publishing from FB) does not work, doesnt retrieve the article or the photo.
I dont know if the rate limiting is the only way to go, any other alternatives?
Thank you !

Other than being annoying, is it causing any problems?

Well, yes, depends on the intensity but sometimes the server load increases very quickly, sometimes is so high that wp stops connecting to the DB. I have to enable the under attack mode for a few minutes, until the bot goes away

I take it you have no page caching? That should take most of the load off of your database.

Thank you for your replies.
I use W3 total cache, I do have the first option enabled (entry pages), I didn’t set the second one (the home/entry page) because it changes a lot (I dont know if enabled refreshes when new posts are published), categories, tags, comments (I dont use comments) it is not set, cache ssl requests it’s on. Catching URL for search it is not set, 404 also not, enabled the last two (local users conected)
On CF I do have in Caching level as Standard and TTL 30 min (using the CF Pro).
Do you mean this two options ? Do you have other suggestions?
thank you!

I suggest you set W3TC to do all caching. It should clear the home page on updates, or it might even have an option to force home page purge. Anything would be better than what’s happening now. And since you don’t use comments, this should be pretty effective.

CF caching is only for static assets, such as images, JS, and CSS. Not pages. You’d probably be better off increasing that TTL.

Looking for more page cache options, In general, I do have Page cache ON, method: Disc enhanced
Database cache it is no set but object cache it is. Browser cache is also on.
Is this ok?
Thank you I will follow your recomendations.

Then that should help with the bot crawling. Is there a TTL setting for that?

mmm no, that option has only Page cache and method,
I dont see a TTL setting in page options or general options.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.