I have been getting sporadic but aggressive hits from Facebook’s bot (facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)). The hits come in rapidfire succession from multiple IP addesses from Irland, the UK, and Brazil.
• 2a03:2880:20ff:f::face:b00c (Ireland)
• 2a03:2880:20ff:13::face:b00c (Ireland)
• 2a03:2880:20ff:25::face:b00c (Ireland)
• 2a03:2880:11ff:18::face:b00c (UK)
• 2a03:2880:11ff:6::face:b00c (UK)
• 2a03:2880:21ff:b::face:b00c (Ireland)
• 2a03:2880:21ff:17::face:b00c (Ireland)
• 2a03:2880:ff:10::face:b00c (Brazil)
• 2a03:2880:32ff:d::face:b00c (Ireland)
This is a very short list from today, and many are hitting very old pages that I cannot imagine are Facebook user previews from entering large numbers of URLs. Moreover, they are all from outside the United States.
I have tried to block this user agent when it comes from outside the US, but it breaks the preview feature on Facebook. I also tried challenge, and that breaks the feature as well.
The behavior looks more like a crawler than users sharing urls and pulling previews.
The bot can become so rapid fire that it will cause the site to start throwing up errors.
Is there a solution to this problem? Can I throttle a user agent independent of the IP address? I have emailed Facebook (as suggested at http://www.facebook.com/externalhit_uatext.php) but have not recieved a reply.