Our website is being hit with about 30-40 of ChatGPT’s crawler bots continuously and all at once for a few days now. We don’t want to block them from scraping, but how can we limit their access to just one of their bots on our site at a time to prevent them from consuming too many of our site server’s resources?
I already created a firewall rule to hit their useragent with a captcha challenge, but it hasn’t slowed them down.
Thank you.