OpenAI bots are overloading the site

Website, Application, Performance
1

What is the name of the domain?

htmag.co.il

What is the error number?

no error

What is the issue you’re encountering

GPT bots are bombarding our site

What steps have you taken to resolve the issue?

I have added a robots.txt with crawl-delay=10

I also tried to block a few types of bots

What are the steps to reproduce the issue?

I can’t seem to find any way to throttle specific bots, I can either set “I’m under attack”, completely block AI bots.

But in this case, I want to throttle the bot (i.e., give the bot 429 errors if it’s running above a certain rate).

2

Sounds like you want to set up rate limiting https://developers.cloudflare.com/rate-limiting/

3

I’m on the pro account. Is there a rate limiting option for pro?

It’s a critical feature now with the rise of ai bots. They are bombarding us and likely many others.

4

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.