I’m the founder of a software which is officially going public this week that browses websites to look for specific content. It which relies on a bot identified by a fixed user agent created for this purpose. The bot identifies and respects robots.txt. CloudFlare is blocking some of my requests and is returning a 403 Forbidden.
I’m currently capping the number of requests per second to 4 for the entire agent, which browses ~3 websites at once, so on average I’m rarely browsing more than 1 page per second per website.
I also think this is partially IP related, as I can see lower rejections when running it from my laptop when compared to running it in AWS. If I remove the user agent, I don’t get rejected requests at all - which is weird since using a user agent is somewhat a need to be compliant with the “bot rules”.
Does anyone have any idea on how I could proceed? Is there a way to contact Cloudflare to whitelist a specific user agent?