How to rate-limit facebookexternalhit Requests

Today we were attacked by 10k+ requests within a minute by official Facebook IPs and all with useragent containing facebookexternalhit.

I want to rate-limit such requests but the WAF does not allow to filter for certain useragents.

How can I achieve this? Facebook is anyways limiting the traffic away from FB so one conclusion could also be that we fully block all requests containing this useragent. Why not?

Rate limiting using fields (such as http.user_agent) is available with a Business Plan…

You could also use the WAF just to block all of these, or use a Worker and make your own rate limiter.

ok, something like this:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request));
})

const RATE_LIMIT = 10; // Number of requests
const TIME_WINDOW = 60 * 1000; // Time window in milliseconds (1 minute)
const BLOCK_DURATION = 5 * 60 * 1000; // Block duration in milliseconds (5 minutes)
const USER_AGENT_BLOCKED = 'BlockedUserAgent'; // User-Agent to be rate limited

async function handleRequest(request) {
  const userAgent = request.headers.get('User-Agent') || '';
  
  if (userAgent.includes(USER_AGENT_BLOCKED)) {
    const clientIP = request.headers.get('CF-Connecting-IP');
    const now = Date.now();
    const rateLimitKey = `rate_limit:${clientIP}`;
    const blockKey = `block:${clientIP}`;

    const blockTimestamp = await getCache(blockKey);
    if (blockTimestamp && (now - blockTimestamp) < BLOCK_DURATION) {
      return new Response('Too many requests, please try again later.', { status: 429 });
    }

    const rateLimitInfo = await getCache(rateLimitKey);
    let requestCount = 0;
    let firstRequestTime = now;

    if (rateLimitInfo) {
      ({ requestCount, firstRequestTime } = JSON.parse(rateLimitInfo));
    }

    if ((now - firstRequestTime) < TIME_WINDOW) {
      requestCount++;
      if (requestCount > RATE_LIMIT) {
        await setCache(blockKey, now, BLOCK_DURATION);
        return new Response('Too many requests, please try again later.', { status: 429 });
      }
    } else {
      requestCount = 1;
      firstRequestTime = now;
    }

    await setCache(rateLimitKey, JSON.stringify({ requestCount, firstRequestTime }), TIME_WINDOW);
  }

  return fetch(request);
}

async function getCache(key) {
  const cache = caches.default;
  const response = await cache.match(key);
  return response ? response.text() : null;
}

async function setCache(key, value, expirationTtl) {
  const cache = caches.default;
  const headers = { 'Content-Type': 'text/plain', 'Cache-Control': `max-age=${expirationTtl}` };
  await cache.put(key, new Response(value, { headers }));
}
1 Like

Hi, Business Plan costs 180$ whereas PRO plan is around 20$. A lot of money for just a little feature. Your proposed solution via Workers is also very expensive for web pages with lots of page views (you pay per views for Workers). So it doesn’t look good for our situation.

2 Likes